[ 
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373407#comment-16373407
 ] 

Hudson commented on PHOENIX-4625:
---------------------------------

SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1820 (See 
[https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1820/])
PHOENIX-4625 memory leak in PhoenixConnection if scanner renew lease (tdsilva: 
rev 4fc3f7545db831e83bc82783a0655df79821c107)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java


> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -----------------------------------------------------------------------------
>
>                 Key: PHOENIX-4625
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4625
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.14.0
>            Reporter: Vikas Vishwakarma
>            Assignee: Vikas Vishwakarma
>            Priority: Major
>             Fix For: 4.14.0, 5.0.0
>
>         Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png
>
>
> We have two different code path
>  # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the 
> following checks  if renew lease feature is supported and if the renew lease 
> config is enabled 
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled
>  # In PhoenixConnection for every scan iterator is added to a Queue for lease 
> renewal based on just the check if the renew lease feature is supported 
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is 
> enabled (phoenix.scanner.lease.renew.enabled)
>  
> Now consider a situation where Renew lease feature is supported but 
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In 
> this case PhoenixConnection will keep adding the iterators for every scan 
> into the scannerQueue for renewal based on the feature supported check but 
> the renewal task is not running because phoenix.scanner.lease.renew.enabled 
> is set to false, so the scannerQueue will keep growing as long as the 
> PhoenixConnection is alive and multiple scans requests are coming on this 
> connection.
>  
> We have a use case that uses a single PhoenixConnection that is perpetual and 
> does billions of scans on this connection. In this case scannerQueue is 
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>  
> Add iterators for Lease renewal in PhoenixConnection
> =============================================
> {code:java}
>  
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
>  if (services.supportsFeature(Feature.RENEW_LEASE))
>  { 
>    checkNotNull(itr); scannerQueue.add(new 
> WeakReference<TableResultIterator>(itr)); 
>  }
> }
> {code}
>  
> Starting the RenewLeaseTask
> =============================
> checks if Feature.RENEW_LEASE is supported and if 
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>  
> ConnectionQueryServicesImpl {
> ....
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, 
> DEFAULT_RENEW_LEASE_ENABLED);
> .....
> @Override
>  public boolean isRenewingLeasesEnabled(){ 
>    return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && 
> renewLeaseEnabled; 
>  }
> private void scheduleRenewLeaseTasks() {
>  if (isRenewingLeasesEnabled()) {
>    renewLeaseExecutor =
>    Executors.newScheduledThreadPool(renewLeasePoolSize, 
> renewLeaseThreadFactory);
>    for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q : 
> connectionQueues) { 
>      renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, 
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
>    }
>   }
> }
> .......
> }
> {code}
>  
> To solve this We must add both checks in PhoenixConnection if the feature is 
> supported and if the config is enabled before adding the iterators to 
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  
> phoenix.scanner.lease.renew.enabled is true 
> instead of just checking if the feature 
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to