[
https://issues.apache.org/jira/browse/IMPALA-12909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892264#comment-17892264
]
Michael Smith commented on IMPALA-12909:
----------------------------------------
I think the piece that restricts DataSourceScanNode to only executing on
coordinator is it's implementation of {{computeScanRangeLocations}} and how
scheduler.cc schedules the scan ranges. That needs some modification to balance
DataSourceScanNode placement across executors.
At a first pass, the ScanRangeLocation should reflect the network address we
actually use to access the data source, so we can co-locate if an executor
happens to exist there. I'm not entirely certain your statement that external
DataSourceScanNodes will only be located on coordinators is true; I haven't yet
found a mechanism that would force that, it might just be a coincidence of
running in the minicluster.
> Generate distributed plan for query accessing multiple JDBC tables
> ------------------------------------------------------------------
>
> Key: IMPALA-12909
> URL: https://issues.apache.org/jira/browse/IMPALA-12909
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Wenzhe Zhou
> Assignee: Pranav Yogi Lodha
> Priority: Major
>
> For a query which access multiple JDBC tables, Planner generate single node
> plan. It's better to generate distributed plan so that JDBC read could be
> scheduled on executors. This restriction is due to current design of External
> data source framework because scan is single threaded. DataSourceScanNode
> cannot run in node other than coordinator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
