[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373502#comment-16373502 ] Hudson commented on PHOENIX-4625: - SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #46 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/46/]) PHOENIX-4625 memory leak in PhoenixConnection if scanner renew lease (tdsilva: rev cb682c9a19695ed33c7e6c3889c20b4071cfa9e7) * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373407#comment-16373407 ] Hudson commented on PHOENIX-4625: - SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1820 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1820/]) PHOENIX-4625 memory leak in PhoenixConnection if scanner renew lease (tdsilva: rev 4fc3f7545db831e83bc82783a0655df79821c107) * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373201#comment-16373201 ] Thomas D'Silva commented on PHOENIX-4625: - Sure I will get this committed today. > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0 > > Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373188#comment-16373188 ] James Taylor commented on PHOENIX-4625: --- [~tdsilva] - would you have some spare cycles to commit this branch to 4.x, master, and 5.x branches? > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0 > > Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372463#comment-16372463 ] Samarth Jain commented on PHOENIX-4625: --- +1 > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0 > > Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372460#comment-16372460 ] Vikas Vishwakarma commented on PHOENIX-4625: Tested locally , attached the GC graphs with the issue (GC_Leak.png) and after fix (GC_After_fix.png) > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0 > > Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372452#comment-16372452 ] James Taylor commented on PHOENIX-4625: --- [~samarthjain] - would you have any spare cycles to review? > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0 > > Attachments: PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372438#comment-16372438 ] Vikas Vishwakarma commented on PHOENIX-4625: [~jamestaylor] please review just replaced this in PhoenixConnection services.supportsFeature(Feature.RENEW_LEASE) with services.isRenewingLeasesEnabled() > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.0 >Reporter: Vikas Vishwakarma >Priority: Major > Fix For: 4.14.0 > > Attachments: PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372424#comment-16372424 ] James Taylor commented on PHOENIX-4625: --- Thanks for the excellent analysis, [~vik.karma]. How about a patch? > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.0 >Reporter: Vikas Vishwakarma >Priority: Major > Attachments: QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { >checkNotNull(itr); scannerQueue.add(new > WeakReference(itr)); > } > } > {code} > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled(){ >return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { >renewLeaseExecutor = >Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); >for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); >} > } > } > ... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled
[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372409#comment-16372409 ] Vikas Vishwakarma commented on PHOENIX-4625: [~larsh] [~giacomotaylor] this is one of the memory leak case observed with QueryServer > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > - > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.0 >Reporter: Vikas Vishwakarma >Priority: Major > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > = > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) { > checkNotNull(itr); > scannerQueue.add(new WeakReference(itr)); > } > } > > Starting the RenewLeaseTask > = > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > ConnectionQueryServicesImpl { > > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > . > @Override > public boolean isRenewingLeasesEnabled() { > return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { > renewLeaseExecutor = > Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); > for (LinkedBlockingQueueq : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); > } > } > } > ... > } > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)