[
https://issues.apache.org/jira/browse/IMPALA-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18048258#comment-18048258
]
ASF subversion and git services commented on IMPALA-12809:
----------------------------------------------------------
Commit 3a5a6f612a332fc509cfdc73c4566356a00ac730 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3a5a6f612 ]
IMPALA-14638: Schedule union of iceberg metadata scanner to coordinator
On clusters with dedicated coordinators and executors the Iceberg
metadata scanner fragment(s) must be scheduled to coordinators.
IMPALA-12809 ensured this for most plans, but if the Iceberg metadata
scanner is part of a union of unpartitioned fragments a new fragment is
created for the union that subsumes existing fragments and loses the
coordinatorOnly flag.
Fixes cases where a multi-fragment plan includes a union of iceberg
metadata scans by setting coordinatorOnly on the new union fragment.
Adds new planner and runtime tests for this case.
Change-Id: If2f19945037b4a7a6433cd9c6e7e2b352fae7356
Reviewed-on: http://gerrit.cloudera.org:8080/23803
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Iceberg metadata table scanner should always be scheduled to the coordinator
> ----------------------------------------------------------------------------
>
> Key: IMPALA-12809
> URL: https://issues.apache.org/jira/browse/IMPALA-12809
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.4.0
> Reporter: Tamas Mate
> Assignee: Daniel Becker
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 4.4.0
>
>
> On larger clusters the Iceberg metadata scanner can be scheduled to
> executors, for example during a join. The fragment in this case will fail a
> precondition check, because either the frontend_ object will not be present
> or the table. Setting {{exec_at_coord}} to true is not enough and these
> fragments should be scheduled to the {{{}coord_only_executor_group{}}}.
> Additionally, setting NUM_NODES=1 should be a viable workaround.
> Reproducible with the following local dev Impala cluster:
> {{./bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=1
> --use_exclusive_coordinators}}
> and query:
> {{select count(b.parent_id) from
> functional_parquet.iceberg_query_metadata.history a}}
> {{join functional_parquet.iceberg_query_metadata.history b on a.snapshot_id =
> b.snapshot_id;}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
