[ 
https://issues.apache.org/jira/browse/HBASE-30544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HBASE-30544:
-----------------------------------
    Labels: pull-request-available  (was: )

> HBaseContext releases connection references before lazy mapPartitions 
> iterators finish
> --------------------------------------------------------------------------------------
>
>                 Key: HBASE-30544
>                 URL: https://issues.apache.org/jira/browse/HBASE-30544
>             Project: HBase
>          Issue Type: Sub-task
>          Components: hbase-connectors
>            Reporter: mazhengxuan
>            Assignee: mazhengxuan
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Impact
> HBaseContext.mapPartitions releases its cached connection reference when the 
> callback returns an Iterator, although that iterator may still use the 
> connection.
> Idle-cache cleanup can therefore close the connection while the task is still 
> consuming results, causing subsequent HBase operations to fail.
> The same pattern exists in the Spark 3 and Spark 4 modules, including 
> delegating RDD/Java wrappers. Spark 3 streamMapPartitions also uses this 
> helper. Runtime reproduction was performed on Spark 4; Spark 3 was checked in 
> source only.
> h3. Reproduction
> Verified with Spark 4.0.0, Scala 2.13.14, and JDK 17, using a simulated 
> cached Connection:
> # Set the internal cache test timeout to 10 ms.
> # Run HBaseContext.mapPartitions on one partition, ensuring only the task's 
> connection reference remains.
> # Return a lazy iterator. In next(), wait 100 ms, invoke normal cleanup with 
> performHousekeeping(false), and check connection.isClosed().
> # Compare with an eager callback that performs the same wait and cleanup 
> before returning its iterator.
> Observed:
> {code}
> eager_connection_closed=false
> lazy_connection_closed=true
> {code}
> Expected: the connection remains open throughout iteration.
> The reduced timeout only accelerates reproduction; the default cache close 
> delay is 10 minutes.
> h3. Suggested fix
> Retain the cached connection reference until task completion, using a 
> TaskCompletionListener.
> Release it exactly once on success, failure, cancellation, or partial 
> iterator consumption. Coordinate this with the existing finally block to 
> prevent double release, and apply the fix to both connector modules.
> Add regression tests covering lazy iteration across cache expiry and 
> reference cleanup after task completion.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to