Sigma-Ma opened a new pull request, #179:
URL: https://github.com/apache/hbase-connectors/pull/179

   ## What changes were proposed in this pull request?
   
   Update HBaseContext in both the Spark 3 and Spark 4 connectors to retain 
each cached connection reference until task completion.
   
   Register a TaskCompletionListener before invoking the mapPartitions callback 
and remove the callback-scoped finally block. Each borrowed reference is 
released once when its task finishes.
   
   Add 13 lifecycle regression tests per connector, covering lazy iteration 
across cache expiry, callback and iterator failures, partial consumption, empty 
iterators, cancellation, completion-listener failures, multiple borrows in one 
task, and concurrent borrowers.
   
   ## Why are the changes needed?
   
   A mapPartitions callback can return a lazy iterator that continues to access 
HBase after the callback returns. Releasing the connection reference at that 
point allows idle-cache cleanup to close the connection while the task is still 
consuming the iterator.
   
   Retaining the reference until task completion protects lazy iteration and 
provides cleanup for success, failure, cancellation, and partial consumption.
   
   ## How was this patch tested?
   
   ```bash
   mvn -o -B -ntp -pl spark/hbase-spark 
'-Dtest=TestJavaHBaseContext#testBulkGet' 
-Dsuites=org.apache.hadoop.hbase.spark.HBaseContextConnectionLifetimeSuite,org.apache.hadoop.hbase.spark.HBaseConnectionCacheSuite,org.apache.hadoop.hbase.spark.HBaseContextSuite,org.apache.hadoop.hbase.spark.HBaseRDDFunctionsSuite
 -Dsurefire.failIfNoSpecifiedTests=true -Dsurefire.rerunFailingTestsCount=0 test
   
   mvn -o -B -ntp -Pspark4 -pl spark4/hbase-spark4 
-Dsuites=org.apache.hadoop.hbase.spark.HBaseContextConnectionLifetimeSuite,org.apache.hadoop.hbase.spark.HBaseConnectionCacheSuite,org.apache.hadoop.hbase.spark.HBaseContextSuite
 -Dsurefire.rerunFailingTestsCount=0 test
   
   mvn -o -B -ntp -Pspark4 -pl spark/hbase-spark,spark4/hbase-spark4 
'-DspotlessFiles=.*/HBaseContext.scala|.*/HBaseContextConnectionLifetimeSuite.scala'
 spotless:check
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to