Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22500 )
Change subject: IMPALA-13768: Redundant Iceberg delete records are shuffled around which cause error "Invalid file path arrived at builder" ...................................................................... IMPALA-13768: Redundant Iceberg delete records are shuffled around which cause error "Invalid file path arrived at builder" IcebergDeleteBuilder assumes that it should only receive delete records for paths of data files that are scheduled for its corresponding SCAN operator. It is not true when any of the following happens: * number of output channels in sender is 1 (currently no DIRECTED mode, no filtering) * hit bug in DIRECTED mode, see below * single node plan is being used (no DIRECTED mode, no filtering) With this patch, KrpcDataStreamSender::Send() will use DIRECTED mode even if number of output channels is 1. It also fixes the bug in DIRECTED mode (which was due to an unused variable 'skipped_prev_row') and simplified the logic a bit. The patch also relaxes the assumption in IcebergDeleteBuilder, i.e. only return error for dangling delete records when we are in a distributed plan where we can assume DIRECTED distribution mode of position delete records. Testing * added e2e tests Change-Id: I695c919c9a74edec768e413a02b2ef7dbfa0d6a5 Reviewed-on: http://gerrit.cloudera.org:8080/22500 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/iceberg-delete-builder.cc M be/src/runtime/krpc-data-stream-sender.cc M testdata/workloads/functional-query/queries/QueryTest/iceberg-delete-partitioned.test 3 files changed, 51 insertions(+), 22 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/22500 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I695c919c9a74edec768e413a02b2ef7dbfa0d6a5 Gerrit-Change-Number: 22500 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
