Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18536 )
Change subject: IMPALA-10465: Use IGNORE variant of Kudu write operations ...................................................................... IMPALA-10465: Use IGNORE variant of Kudu write operations KUDU-1563 added support for INSERT_IGNORE, UPDATE_IGNORE, and DELETE_IGNORE to handle cases where users want to ignore primary key errors efficiently. Impala already does this today for its INSERT behavior. However, it does so by ignoring the per-row errors from Kudu client side. This requires a large error buffer (which may need to be expanded in rare cases) to log all of the warning messages which users often do not care about and causes significant RPC overhead. This patch change the Kudu write operation by Impala to use INSERT_IGNORE, UPDATE_IGNORE, and DELETE_IGNORE if Kudu cluster supports it and backend flag "kudu_ignore_conflicts" is true. We benchmark the change by doing insert and update query on modified tpch.lineitem table where we introduce conflicts for around half of the total rows being modified. The table below shows the performance difference after the patch: +----------------------+--------+-------------+------------+------------+----------------+ | Query | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | +----------------------+--------+-------------+------------+------------+----------------+ | KUDU-IGNORE-3-UPDATE | 30.06 | 30.52 | -1.53% | 0.18% | 0.58% | | KUDU-IGNORE-2-INSERT | 48.91 | 71.09 | I -31.20% | 0.60% | 0.72% | +----------------------+--------+-------------+------------+------------+----------------+ Testing: - Pass core tests. Change-Id: I8da7c41d61b0888378b390b8b643238433eb3b52 Reviewed-on: http://gerrit.cloudera.org:8080/18536 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java M tests/custom_cluster/test_kudu.py 4 files changed, 380 insertions(+), 36 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18536 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8da7c41d61b0888378b390b8b643238433eb3b52 Gerrit-Change-Number: 18536 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
