[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: 2.6.0 (was: 2.5.0) > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 1.8.0, 3.0.0-alpha-2, 2.6.0 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: (was: 2.4.6) > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Assignee: Prathyusha >Priority: Major > Fix For: 2.5.0, 1.8.0, 3.0.0-alpha-2 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: (was: 2.4.5) 2.4.6 > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Assignee: Prathyusha >Priority: Major > Fix For: 2.5.0, 1.8.0, 3.0.0-alpha-2, 2.4.6 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: (was: 2.4.4) 2.4.5 > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Assignee: Prathyusha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 1.8.0, 2.4.5 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: (was: 2.4.3) 2.4.4 > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Assignee: Prathyusha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 1.8.0, 2.4.4 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-25624: -- Fix Version/s: (was: 1.7.0) 1.8.0 > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Assignee: Prathyusha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 1.8.0, 2.4.3 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Description: We have a large table in production that causes the balancer's RegionLocationFinder cache to consume 4 GB of heap, which, among other factors, triggered OOMEs, and made us aware of this problem. RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The RegionLocationFinder cache comes to consume heap for RegionInfos for all table regions and all HDFS block locations of all store files for all regions of all tables. The only limit we pass to the CacheBuilder is an expiration time of 1440 milliseconds for individual cache entries. That's 4 hours. That's much too long; however, the cache also periodically refreshes itself, where the need for a refresh is checked whenever BaseLoadBalancer calls RegionLocationFinder's setClusterMetrics() method, which defeats the expiration based limit anyway. We should be bounding this cache with effective resource controls. Time based expiry is fine but the periodic refresh logic must be removed to make it effective. Implement size based limits too. CacheBuilder#maximumSize will limit by number cache entries. This might be fine but CacheBuilder#maximumWeight would be better, where weight is something determined by the API user. In this case it can be an estimate of the heap size of the hash map entries kept in the cache. Default should remain unbounded. Specific bounds should be supported by new site configuration options. was: We have a large table in production that causes the balancer's RegionLocationFinder cache to consume 4 GB of heap, which, among other factors, triggered OOMEs, and made us aware of this problem. RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The RegionLocationFinder cache comes to consume heap for RegionInfos for all table regions and all HDFS block locations of all store files for all regions of all tables. The only limit we pass to the CacheBuilder is an expiration time of 1440 milliseconds for individual cache entries. That's 4 hours. That's much too long; however, the cache also periodically refreshes itself, where the need for a refresh is checked whenever BaseLoadBalancer calls RegionLocationFinder's setClusterMetrics() method, which defeats the expiration based limit anyway. We should be bounding this cache with effective resource controls. Time based expiry is fine but the periodic refresh logic must be removed to make it effective. Implement size based limits too. CacheBuilder#maximumSize will limit by number cache entries. This might be fine but CacheBuilder#maximumWeight would be better, where weight is something determined by the API user. In this case it can be an estimate of the heap size of the hash map entries kept in the cache. > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.3 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
