Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18536
Change subject: IMPALA-10465: Use IGNORE variant of Kudu write operations ...................................................................... IMPALA-10465: Use IGNORE variant of Kudu write operations KUDU-1563 added support for INSERT_IGNORE, UPDATE_IGNORE, and DELETE_IGNORE to handle cases where users want to ignore primary key errors in an efficient way. Impala already does this today for it's INSERT behavior, however it does so by ignoring the per-row errors from Kudu client side. This requires a large error buffer (which may need to be expanded in rare cases) to log all of the warning messages which users often do not care about and causes significant RPC overhead. This patch change the the Kudu write operation by Impala to use INSERT_IGNORE, UPDATE_IGNORE, and DELETE_IGNORE if Kudu master server support it and backend flag "kudu_ignore_conflicts" is true. The table below shows performance difference after the patch by runing kudu-ignore workload in 9 iterations: +----------------------+--------+-------------+------------+------------+----------------+ | Query | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | +----------------------+--------+-------------+------------+------------+----------------+ | KUDU-IGNORE-0-CREATE | 1.92 | 1.70 | +13.01% | * 48.54% * | * 37.55% * | | KUDU-IGNORE-3-DELETE | 0.06 | 0.06 | +1.21% | 1.61% | 1.33% | | KUDU-IGNORE-2-UPDATE | 0.16 | 0.16 | -0.58% | 0.54% | 2.22% | | KUDU-IGNORE-1-INSERT | 0.13 | 0.20 | I -34.19% | * 16.25% * | * 13.03% * | +----------------------+--------+-------------+------------+------------+----------------+ Testing: - Pass core tests. Change-Id: I8da7c41d61b0888378b390b8b643238433eb3b52 --- M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M bin/single_node_perf_run.py M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java A testdata/workloads/targeted-perf/queries/kudu-ignore.test M tests/custom_cluster/test_kudu.py 6 files changed, 123 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/18536/1 -- To view, visit http://gerrit.cloudera.org:8080/18536 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8da7c41d61b0888378b390b8b643238433eb3b52 Gerrit-Change-Number: 18536 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <[email protected]>
