Tim Armstrong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16029 )
Change subject: IMPALA-9782: fix Kudu DML with mt_dop ...................................................................... IMPALA-9782: fix Kudu DML with mt_dop In general, ScalarExpr subclasses should not have mutable state, i.e. state that is modified during query execution, and definitely need to be thread-safe. KuduPartitionExpr violated that, since KuduPartitioner and KuduPartialRow are both mutated as part of expr evaluation. This patch moves those objects into thread-local state for KuduPartitionExpr. Testing: * Add a regression test that does a large insert. Before this patch it reliably crashed Impala with mt_dop=4. * Run more Kudu DML with mt_dop to improve test coverage. * Make test_kudu.py use the correct test dimension (fixing a cosmetic issue). Perf: This changes adds some indirection to the expression evaluation, so I did some manual benchmarking with this query, which should somewhat stress the partitioning: set mt_dop=1; insert into orders_key_only select o_orderkey from tpch_kudu.orders where o_orderkey % 10 = 0 The timing was in the same range before and after - between 6 and 8 seconds - but the results were very unstable so inconclusive. The Kudu tservers were using an order-of-magnitude more CPU than the impalads, so it seems safe to conclude that these partitioning exprs are *not* a bottleneck for DML workloads. Perf record showed impala::KuduPartitionExpr::GetIntValInterpreted() taking up 0.09% of the impalad CPU, for additional evidence that this doesn't make a difference. Change-Id: Ie7e86894da9dbcb3b69f7db236c841ecc08dbf1a --- M be/src/exprs/kudu-partition-expr.cc M be/src/exprs/kudu-partition-expr.h M be/src/exprs/scalar-expr-evaluator.h M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M tests/common/kudu_test_suite.py M tests/query_test/test_kudu.py 6 files changed, 132 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/16029/2 -- To view, visit http://gerrit.cloudera.org:8080/16029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie7e86894da9dbcb3b69f7db236c841ecc08dbf1a Gerrit-Change-Number: 16029 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
