[
https://issues.apache.org/jira/browse/IMPALA-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089839#comment-18089839
]
ASF subversion and git services commented on IMPALA-14523:
----------------------------------------------------------
Commit cffe56518feb0a899c9cee9dd7bdb10f96106abc in impala's branch
refs/heads/master from Arnab Karmakar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=cffe56518 ]
IMPALA-15076: Fix JDBC scan duplication in UNION ALL and ResultSet race
After IMPALA-14523 patch (shared JDBC cursor fetching), UNION ALL
queries with JDBC tables produced incorrect results due to
duplicate table scans. The scheduler assigned each JDBC scan node to a
different executor based on node_id, causing UNPARTITIONED union
fragments to spawn instances on multiple executors. Each instance would
execute all scan nodes in the union, but only one had assigned ranges,
leading to 3x result duplication in queries.
The fix changes JDBC executor selection to hash fragment_idx instead of
node_id, ensuring all JDBC scans within a union fragment are co-located
on the same executor. This preserves the shared connection optimization
while preventing duplicate reads.
Additionally, fixed a race condition in JdbcRecordIterator where multiple
threads could call ResultSet.next() after end-of-stream. Per JDBC spec,
this behavior is vendor-specific for TYPE_FORWARD_ONLY result sets and
may throw SQLException. Added an endOfStream flag guarded by fetchLock
to prevent post-EOS next() calls.
Testing:
- Core TPC-DS JDBC tests now pass
- All tests passed with exhaustive exploration strategy using release build
- Added JDBC UNION ALL planner tests
Change-Id: I60ed011faa2177af67ea681c2cd2967648e4a963
Reviewed-on: http://gerrit.cloudera.org:8080/24464
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Optimize JDBC table for Hive Multistream Driver
> -----------------------------------------------
>
> Key: IMPALA-14523
> URL: https://issues.apache.org/jira/browse/IMPALA-14523
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 4.4.0
> Reporter: Kurt Deschler
> Assignee: Pranav Yogi Lodha
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> HIVE-27872 added multi-stream fetch capabilites to the HS2 JDBC driver which
> facilitates very fast transport of bulk data over JDBC. However, to achieve
> high performance it is necessary for the client to consume data quickly from
> the single-threaded JDBC client. This requires minimizing the (synchronous)
> work done after fetching data from JDBC and performing any expensive
> processing using multiple threads. It is possible to achive this result
> either by fetching data on a single thread and handing it off for consumption
> or using locking to serialize fetching. In either case, the fetch path must
> copy data in an efficent way from the cursor to local memory and defer any
> expensive encoding/decoding/conversion to a multi-threaded codepath.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
