[
https://issues.apache.org/jira/browse/KUDU-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088893#comment-18088893
]
Daojiang Yan commented on KUDU-3779:
------------------------------------
h2. Benchmark plan and results for in-memory metadata replay on encrypted
clusters
Attached: [^KUDU-3779-bench-logs.tar.gz]
h3. Goal
Verify that the in-memory metadata replay fast path introduced by KUDU-3778
delivers the same kind of speedup on *encrypted* clusters, and that
extending it does not regress the existing non-encrypted path.
h3. Environment
|| Item || Value ||
| Build | release |
| Slow tests | {{KUDU_ALLOW_SLOW_TESTS=1}} |
| Test binary | {{log_block_manager-test}} |
| Filter | {{--gtest_filter='*InMemoryReplayStartupBenchmark*'}} |
| Reopens per pass | 5 ({{--startup_benchmark_reopen_times=5}}) |
| Container metadata format | native (LBM) |
| Comparison axis | {{--log_container_metadata_inmem_replay_threshold_bytes}}:
{{0}} (streaming) vs {{64MiB}} (in-memory) |
h4. Host / runtime
Runs inside a Docker container; the container shares the host kernel
and writes test data under {{/tmp}}, which is backed by the container's
overlay filesystem.
|| Item || Value ||
| CPU | Intel(R) Xeon(R) Silver 4216 @ 2.10 GHz, 1 socket / 16 cores / 32
threads |
| Memory | 62 GiB total, ~29 GiB available as buff/cache during the run |
| Host kernel | Linux 4.18.0-553.47.1.el8_10.x86_64 |
| Container userspace | CentOS Linux 7 (Core) |
| Filesystem at /tmp | overlay (Docker), ~3.7 TiB capacity |
| Underlying disk | sdb, 3.7 TiB, rotational HDD (rota=1); host also has a
223.6 GiB SSD (sda) but it is not where /tmp lives |
| Containerized | Docker |
h3. Test groups
Four groups, mirroring the layout of the KUDU-3778 commit description.
h4. Group A: workload-scale sweep (encryption enabled)
Holds deletion at 90%, varies the total number of records and live blocks.
|| Test || batches x blocks/batch || dirs || live blocks || log file ||
| A_small | 200 x 500 | 4 | 10k | A_small_enc.log |
| A_medium | 500 x 1000 | 8 | 50k | A_medium_enc.log |
| A_large | 1000 x 1000 | 8 | 100k | A_large_enc.log |
h4. Group B: deletion-ratio sweep (encryption enabled)
Holds workload at 500 x 1000 / 8 dirs, varies
{{--startup_benchmark_deleted_block_percentage}}.
|| Test || deletion % || live blocks || log file ||
| B_del0 | 0% | 500k | B_del0_enc.log |
| B_del50 | 50% | 250k | B_del50_enc.log |
| B_del90 | 90% | 50k | B_del90_enc.log |
| B_del99 | 99% | 5k | B_del99_enc.log |
h4. Group C: container-density sweep (encryption enabled)
Holds workload at 500 x 1000 / 8 dirs / 90% deletion, varies
{{--log_container_max_blocks}}. The first reopen of each pass is
excluded from the steady-state mean because it is dominated by
cold page cache.
|| Test || log_container_max_blocks || containers || log file ||
| C_maxb-1 | -1 | 3 | C_maxb-1_enc.log |
| C_maxb10000 | 10000 | 51 | C_maxb10000_enc.log |
| C_maxb1000 | 1000 | 367 | C_maxb1000_enc.log |
h4. Group D: non-encryption regression check
Same workload as A_medium / C_maxb-1 (500 x 1000, 8 dirs, 90% deletion)
but encryption *disabled*. Compared against KUDU-3778 numbers for the
same configuration to confirm no regression.
|| Test || encryption || log file ||
| D_regression | disabled | D_regression_unenc.log |
h3. Reproduce
{code}
Group A:
# A-small: 200×500, 4 dirs (~10k live)
KUDU_ALLOW_SLOW_TESTS=1 build/latest/bin/log_block_manager-test \
--gtest_filter='*InMemoryReplayStartupBenchmark/1' \
--startup_benchmark_reopen_times=5 \
--startup_benchmark_batch_count_for_testing=200 \
--startup_benchmark_block_count_per_batch_for_testing=500 \
--startup_benchmark_data_dir_count_for_testing=4 \
--startup_benchmark_deleted_block_percentage=90 \
--logtostderr 2>&1 | tee log/bench-3779/A_small_enc.log
# A-medium: 500×1000, 8 dirs (~50k live)
KUDU_ALLOW_SLOW_TESTS=1 build/latest/bin/log_block_manager-test \
--gtest_filter='*InMemoryReplayStartupBenchmark/1' \
--startup_benchmark_reopen_times=5 \
--startup_benchmark_batch_count_for_testing=500 \
--startup_benchmark_block_count_per_batch_for_testing=1000 \
--startup_benchmark_data_dir_count_for_testing=8 \
--startup_benchmark_deleted_block_percentage=90 \
--logtostderr 2>&1 | tee log/bench-3779/A_medium_enc.log
# A-large: 1000×1000, 8 dirs (~101k live)
KUDU_ALLOW_SLOW_TESTS=1 build/latest/bin/log_block_manager-test \
--gtest_filter='*InMemoryReplayStartupBenchmark/1' \
--startup_benchmark_reopen_times=5 \
--startup_benchmark_batch_count_for_testing=1000 \
--startup_benchmark_block_count_per_batch_for_testing=1000 \
--startup_benchmark_data_dir_count_for_testing=8 \
--startup_benchmark_deleted_block_percentage=90 \
--logtostderr 2>&1 | tee log/bench-3779/A_large_enc.log
Group B:
for p in 0 50 90 99; do
KUDU_ALLOW_SLOW_TESTS=1 build/latest/bin/log_block_manager-test \
--gtest_filter='*InMemoryReplayStartupBenchmark/1' \
--startup_benchmark_reopen_times=5 \
--startup_benchmark_batch_count_for_testing=500 \
--startup_benchmark_block_count_per_batch_for_testing=1000 \
--startup_benchmark_data_dir_count_for_testing=8 \
--startup_benchmark_deleted_block_percentage=$p \
--logtostderr 2>&1 | tee log/bench-3779/B_del${p}_enc.log
done
Group C:
for m in -1 10000 1000; do
KUDU_ALLOW_SLOW_TESTS=1 build/latest/bin/log_block_manager-test \
--gtest_filter='*InMemoryReplayStartupBenchmark/1' \
--startup_benchmark_reopen_times=5 \
--startup_benchmark_batch_count_for_testing=500 \
--startup_benchmark_block_count_per_batch_for_testing=1000 \
--startup_benchmark_data_dir_count_for_testing=8 \
--startup_benchmark_deleted_block_percentage=90 \
--log_container_max_blocks=$m \
--logtostderr 2>&1 | tee log/bench-3779/C_maxb${m}_enc.log
done
Group D:
KUDU_ALLOW_SLOW_TESTS=1 build/latest/bin/log_block_manager-test \
--gtest_filter='*InMemoryReplayStartupBenchmark/0' \
--startup_benchmark_reopen_times=5 \
--startup_benchmark_batch_count_for_testing=500 \
--startup_benchmark_block_count_per_batch_for_testing=1000 \
--startup_benchmark_data_dir_count_for_testing=8 \
--startup_benchmark_deleted_block_percentage=90 \
--logtostderr 2>&1 | tee log/bench-3779/D_regression_unenc.log
{code}
{{*/1}} selects the {{EncryptionEnabled}} parametrization; use {{*/0}}
for the non-encryption baseline (Group D).
h3. Results (5-reopen mean, ms; container-density group is steady-state)
|| Group || Test || streaming || in-memory || speedup ||
| A | A_small | 202 | 115 | 1.76x |
| A | A_medium | 1016 | 604 | 1.68x |
| A | A_large | 2091 | 1218 | 1.72x |
| B | B_del0 | 703 | 468 | 1.50x |
| B | B_del50 | 899 | 549 | 1.64x |
| B | B_del90 | 1034 | 578 | 1.79x |
| B | B_del99 | 1042 | 577 | 1.81x |
| C | C_maxb-1 | 1018 | 577 | 1.76x |
| C | C_maxb10000 | 54 | 34 | 1.59x |
| C | C_maxb1000 | 28 | 22 | 1.27x |
| D | D_regression (non-encrypted) | 805 | 381 | 2.11x |
h3. Conclusions
* The fast path delivers a 1.50x - 1.81x speedup on encrypted clusters
across all workloads (~40% wall-clock reduction per reopen).
* Group D matches KUDU-3778's reported numbers for the same
configuration (811 ms -> 394 ms in KUDU-3778 vs 805 ms -> 381 ms
here), so the non-encryption path is not regressed.
* The streaming-vs-in-memory shape is identical to KUDU-3778:
streaming time scales with total records, in-memory time only scales
with the per-record decrypt work that cannot be avoided.
* Absolute speedup on encrypted clusters (~1.7x) is lower than on
non-encrypted clusters (~2.1x) because per-record decryption work
has to run on both paths; the fast path can only collapse the
O(records) preadv() syscalls into one bulk read per container.
> Support encrypted metadata in the log block container in-memory replay path
> ---------------------------------------------------------------------------
>
> Key: KUDU-3779
> URL: https://issues.apache.org/jira/browse/KUDU-3779
> Project: Kudu
> Issue Type: Improvement
> Reporter: Daojiang Yan
> Assignee: Daojiang Yan
> Priority: Minor
> Attachments: KUDU-3779-bench-logs.tar.gz
>
>
> KUDU-3778 change introducing
> --log_container_metadata_inmem_replay_threshold_bytes) added an in-memory
> replay fast path for LogBlockContainerNativeMeta::ProcessRecords(): instead
> of issuing two preadv() syscalls per record, the entire metadata file is
> slurped into memory once and ReadablePBContainerFile then reads from a
> MemoryReadableFile wrapper. This drastically reduces syscalls during the
> "Reading filesystem" stage of tserver startup on hosts with many containers.
> The current implementation intentionally falls back to the streaming path
> whenever the metadata file is encrypted (i.e.
> RandomAccessFile::GetEncryptionHeaderSize() > 0). As a result, encrypted
> clusters do not benefit from the optimization.
> *Why encrypted metadata is excluded today*
> ReadablePBContainerFile recovers from an incomplete trailing record (typical
> after a crash mid-write, when the on-disk tail is a file-hole filled with
> zeros) by relying on a per-slice optimization inside DoDecryptV(): it detects
> all-zero ciphertext slices and leaves them as zeros in plaintext rather than
> running them through AES. The PB container parser then sees a zeroed tail and
> returns Status::Incomplete, which the caller handles by truncating the
> partial record.
> If we preload the whole payload as a single buffer and serve all subsequent
> reads from memory, the decryption layer sees one bulk slice instead of
> per-record slices. The trailing zero region is no longer aligned with a slice
> boundary, the zero-slice shortcut is defeated, and the file-hole tail
> decrypts to garbled plaintext. The PB parser would then misclassify a benign
> crash-truncation as corruption.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
