[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-22 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373502#comment-16373502
 ] 

Hudson commented on PHOENIX-4625:
-

SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #46 (See 
[https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/46/])
PHOENIX-4625 memory leak in PhoenixConnection if scanner renew lease (tdsilva: 
rev cb682c9a19695ed33c7e6c3889c20b4071cfa9e7)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java


> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0, 5.0.0
>
> Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-22 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373407#comment-16373407
 ] 

Hudson commented on PHOENIX-4625:
-

SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1820 (See 
[https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1820/])
PHOENIX-4625 memory leak in PhoenixConnection if scanner renew lease (tdsilva: 
rev 4fc3f7545db831e83bc82783a0655df79821c107)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java


> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0, 5.0.0
>
> Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-22 Thread Thomas D'Silva (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373201#comment-16373201
 ] 

Thomas D'Silva commented on PHOENIX-4625:
-

Sure I will get this committed today.

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-22 Thread James Taylor (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373188#comment-16373188
 ] 

James Taylor commented on PHOENIX-4625:
---

[~tdsilva] - would you have some spare cycles to commit this branch to 4.x, 
master, and 5.x branches?

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-21 Thread Samarth Jain (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372463#comment-16372463
 ] 

Samarth Jain commented on PHOENIX-4625:
---

+1

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-21 Thread Vikas Vishwakarma (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372460#comment-16372460
 ] 

Vikas Vishwakarma commented on PHOENIX-4625:


Tested locally , attached the GC graphs with the issue (GC_Leak.png) and after 
fix (GC_After_fix.png)

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-21 Thread James Taylor (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372452#comment-16372452
 ] 

James Taylor commented on PHOENIX-4625:
---

[~samarthjain] - would you have any spare cycles to review?

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-21 Thread Vikas Vishwakarma (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372438#comment-16372438
 ] 

Vikas Vishwakarma commented on PHOENIX-4625:


[~jamestaylor] please review

just replaced this in PhoenixConnection

services.supportsFeature(Feature.RENEW_LEASE) with 
services.isRenewingLeasesEnabled()

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.14.0
>Reporter: Vikas Vishwakarma
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-21 Thread James Taylor (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372424#comment-16372424
 ] 

James Taylor commented on PHOENIX-4625:
---

Thanks for the excellent analysis, [~vik.karma]. How about a patch?

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
>Reporter: Vikas Vishwakarma
>Priority: Major
> Attachments: QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>checkNotNull(itr); scannerQueue.add(new 
> WeakReference(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>renewLeaseExecutor =
>Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>for (LinkedBlockingQueue q : 
> connectionQueues) { 
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>}
>   }
> }
> ...
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled

2018-02-21 Thread Vikas Vishwakarma (JIRA)

[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372409#comment-16372409
 ] 

Vikas Vishwakarma commented on PHOENIX-4625:


[~larsh] [~giacomotaylor] this is one of the memory leak case observed with 
QueryServer 

> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
>Reporter: Vikas Vishwakarma
>Priority: Major
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE)) {
>  checkNotNull(itr);
> scannerQueue.add(new WeakReference(itr));
>  }
>  }
>  
> Starting the RenewLeaseTask
> =
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> ConnectionQueryServicesImpl {
> 
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .
> @Override
>  public boolean isRenewingLeasesEnabled() {
>  return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled;
>  }
> private void scheduleRenewLeaseTasks() {
> if (isRenewingLeasesEnabled()) {
> renewLeaseExecutor =
>  Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>  for (LinkedBlockingQueue q : 
> connectionQueues) {
>  renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
>  renewLeaseTaskFrequency, TimeUnit.MILLISECONDS);
>  }
>  }
>  }
> ...
> }
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)