Matthew Jacobs has posted comments on this change. Change subject: IMPALA-3742: Partitions and sort INSERTs for Kudu tables ......................................................................
Patch Set 4: > > > Perf results from running on the 10 node cluster: > > > > > > For smaller queries that we were able to handle previously, > > there's > > > a regression of about 10% in overall query running time (all > > > averaged over 3 runs): > > > 240.13s with the patch vs. 224.58s previously for 200m rows > > > inserted > > > 472.97s with the patch vs. 433.05s previously for 400m rows > > > inserted > > > That's unfortunate, but can be improved in the future, e.g. by > > > codegen-ing the partition function. > > > > > > But, we can now handle significantly larger inserts - I was > > seeing > > > timeouts regularly at > 400m rows previously, but with the > patch > > > I've tested up to 6b row inserts without any timeouts. > > > > Those are good results, and the perf regression is fairly > > negligible, so let's not worry about that for now. > > Currently KuduPartitionExpr::GetIntVal is very expensive, although > it is not currently the bottleneck we should. > During the conducted tests is soft memory limit reached? While I think we should look at what we can do to improve the perf, I don't think this should prevent us from getting this change in. The most important thing is that non-trivial DML statements actually complete without timing out. Nobody will touch this otherwise. We can work on perf after this change, but probably the issue is lack of codegen in the kudu client itself which will be really hard to address and there is probably much lower hanging fruit to reclaim the 11%. -- To view, visit http://gerrit.cloudera.org:8080/6559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I84ce0032a1b10958fdf31faef225372c5c38fdc4 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall <[email protected]> Gerrit-Reviewer: Dimitris Tsirogiannis <[email protected]> Gerrit-Reviewer: Marcel Kornacker <[email protected]> Gerrit-Reviewer: Matthew Jacobs <[email protected]> Gerrit-Reviewer: Mostafa Mokhtar <[email protected]> Gerrit-Reviewer: Thomas Tauber-Marshall <[email protected]> Gerrit-HasComments: No
