Hello Marton Greber, Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24615
to look at the new patch set (#3).
Change subject: [tablet] Skip stable_sort for already sorted batch
......................................................................
[tablet] Skip stable_sort for already sorted batch
Tablet::BulkCheckPresence unconditionally runs std::stable_sort over the
encoded-key/index pairs for every write batch before deduplicating and
issuing the RowSetTree lookup. When a client submits rows already in
increasing key order such as bulk-ingest tools (e.g. 'kudu perf loadgen,
Impala INSERT ... ORDER BY, Spark/Sqoop pipelines that pre-sort by PK)
and workloads with monotonic primary keys typically do, this sort is
pure overhead: the input is already a valid stable-sorted output.
This patch tracks an 'is_sorted' flag while building 'keys_and_indexes'
in the loop that already scans every op, flipping it to false on the
first out-of-order pair. When the flag survives to the end of the loop,
the stable_sort is skipped entirely, improving performance by a slight
margin if not significant.
Measured with 'kudu perf loadgen --use_upsert --num_rows_per_thread=5M'
on a single-tablet cluster (RELEASE build), median of N=5 runs:
Without this patch and sorted input batch:
for i in 1 2 3 4 5; do ./build/latest/bin/kudu perf loadgen \
127.0.0.1:8764 --use_upsert --num_threads=1 \
--num_rows_per_thread=5000000 --table_num_hash_partitions=1 \
--table_num_range_partitions=1 --buffer_flush_watermark_pct=1.0 \
--run_scan=false \
2>&1 | grep "time total"; done
time total: 6620.12 ms
time total: 6590.24 ms
time total: 6663.81 ms
time total: 6717.08 ms
time total: 6713.18 ms
Median: 6663.81 ms
With this patch and sorted input batch:
for i in 1 2 3 4 5; do ./build/latest/bin/kudu perf loadgen \
127.0.0.1:8764 --use_upsert --num_threads=1 \
--num_rows_per_thread=5000000 --table_num_hash_partitions=1 \
--table_num_range_partitions=1 --buffer_flush_watermark_pct=1.0 \
--run_scan=false \
2>&1 | grep "time total"; done
time total: 6383.22 ms
time total: 6440.69 ms
time total: 6548.54 ms
time total: 6586.15 ms
time total: 6600.53 ms
Median: 6548.54 ms
Change-Id: I28d3f5ebdf7e3f13f04ffe8d35417a98c370fcf7
---
M src/kudu/tablet/tablet-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
4 files changed, 259 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/24615/3
--
To view, visit http://gerrit.cloudera.org:8080/24615
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I28d3f5ebdf7e3f13f04ffe8d35417a98c370fcf7
Gerrit-Change-Number: 24615
Gerrit-PatchSet: 3
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
