Michael Smith created IMPALA-15358:
--------------------------------------
Summary: Cap writers to Kudu partitions
Key: IMPALA-15358
URL: https://issues.apache.org/jira/browse/IMPALA-15358
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Michael Smith
Assignee: Michael Smith
For INSERT/UPSERT/UPDATE/DELETE into a partitioned Kudu table, the planner
inserts a KUDU-partitioned exchange ahead of the KuduTableSink. At runtime,
KrpcDataStreamSender routes each row to channel (partition_index %
num_channels), where partition_index is the actual Kudu tablet index (bounded
by the table's tablet count). The writer fragment's instance count, however, is
sized like any other fragment (mt_dop / cost-based scaling) with no awareness
of the target table's partitioning. Whenever the scheduler assigns more writer
instances than the table has tablets, every instance index >= tablet count
never receives rows, causing severe work skew.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
