Hello Marton Greber, Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24615
to look at the new patch set (#4).
Change subject: [tablet] Skip stable_sort for already sorted batch
......................................................................
[tablet] Skip stable_sort for already sorted batch
Tablet::BulkCheckPresence unconditionally runs std::stable_sort over the
encoded-key/index pairs for every write batch before deduplicating and
issuing the RowSetTree lookup. When a client submits rows already in
increasing key order such as bulk-ingest tools (e.g. 'kudu perf loadgen,
Impala INSERT ... ORDER BY, Spark/Sqoop pipelines that pre-sort by PK)
and workloads with monotonic primary keys typically do, this sort is
pure overhead: the input is already a valid stable-sorted output.
This patch tracks an 'is_sorted' flag while building 'keys_and_indexes'
in the loop that already scans every op, flipping it to false on the
first out-of-order pair. When the flag survives to the end of the loop,
the stable_sort is skipped entirely, improving performance by a slight
margin if not significant.
Measured with 'kudu perf loadgen --use_upsert --num_rows_per_thread=5M'
on a single-tablet cluster (RELEASE build), median of N=5 runs:
Without this patch and sorted input batch:
for i in 1 2 3 4 5; do ./build/latest/bin/kudu perf loadgen \
127.0.0.1:8764 --use_upsert --num_threads=1 \
--num_rows_per_thread=5000000 --table_num_hash_partitions=1 \
--table_num_range_partitions=1 \
--buffer_size_bytes=$((96 * 1024 * 1024)) \
2>&1 | grep "time total"; done
time total: 4419.16 ms
time total: 4433.7 ms
time total: 4430.2 ms
time total: 4395.23 ms
time total: 4346.59 ms
Median: 4419 ms
Mean: 4405 ms
With this patch and sorted input batch:
for i in 1 2 3 4 5; do ./build/latest/bin/kudu perf loadgen \
127.0.0.1:8764 --use_upsert --num_threads=1 \
--num_rows_per_thread=5000000 --table_num_hash_partitions=1 \
--table_num_range_partitions=1 \
--buffer_size_bytes=$((96 * 1024 * 1024)) \
2>&1 | grep "time total"; done
time total: 4181.83 ms
time total: 4103.07 ms
time total: 4183.71 ms
time total: 4121.75 ms
time total: 4133.22 ms
Median: 4133 ms
Mean: 4145 ms
Median speedup: 6.5%
Mean speedup: 5.9%
Change-Id: I28d3f5ebdf7e3f13f04ffe8d35417a98c370fcf7
---
M src/kudu/tablet/tablet-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
4 files changed, 259 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/24615/4
--
To view, visit http://gerrit.cloudera.org:8080/24615
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I28d3f5ebdf7e3f13f04ffe8d35417a98c370fcf7
Gerrit-Change-Number: 24615
Gerrit-PatchSet: 4
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
