[
https://issues.apache.org/jira/browse/KUDU-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092638#comment-18092638
]
ASF subversion and git services commented on KUDU-3779:
-------------------------------------------------------
Commit 8582d9a6429c2ef1a38b0d33bfa86db8efdb4656 in kudu's branch
refs/heads/master from Yan-Daojiang
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=8582d9a64 ]
KUDU-3779: support in-memory metadata replay for encrypted clusters
KUDU-3778 (Change-Id: Iacbe12977ae945e) sped up tserver startup by
reading each LBM native-meta container's .metadata file fully into
memory before replay, but disabled the optimization on encrypted
clusters: the in-memory shim bypassed the env's per-Read decryption,
and a bulk decrypt at preload time would have broken the per-slice
IsAllZeros() short-circuit that KUDU-2260 trailing-zero recovery
depends on.
This change extends the optimization to encrypted clusters by caching
the ciphertext in memory and decrypting on each in-memory Read().
RandomAccessFile gains two virtuals with no-op defaults
(ReadRaw / Decrypt) so the non-encrypted path is unchanged;
PosixRandomAccessFile overrides them to expose its existing
DoReadV / DoDecryptV helpers, which keep the IsAllZeros() short-circuit
intact. MaybePreloadMetadataIntoMemory() no longer gates on encryption
and applies the existing replay-threshold flag to the payload size.
Non-encrypted behavior is bit-for-bit unchanged. Encrypted clusters
now get the same syscall collapse (O(records) -> O(containers)) and
still recover trailing zero pads correctly.
Benchmarks (release build, KUDU_ALLOW_SLOW_TESTS=1,
`log_block_manager-test --gtest_filter='*InMemoryReplayStartupBenchmark*'`,
5 reopens per pass, native-meta containers, encryption ENABLED):
"streaming" = --log_container_metadata_inmem_replay_threshold_bytes=0,
"in-memory" = --log_container_metadata_inmem_replay_threshold_bytes=64MiB.
Workload scale (batches x blocks/batch, dirs, 90% deletion):
200 x 500, 4 dirs ( 10k live): 202 ms -> 115 ms (1.76x, -43%)
500 x 1000, 8 dirs ( 50k live): 1016 ms -> 604 ms (1.68x, -41%)
1000 x 1000, 8 dirs (100k live): 2091 ms -> 1218 ms (1.72x, -42%)
Deletion-ratio sweep (500x1000 / 8 dirs, encryption enabled):
deleted=0% (500k records, 500k live): 703 ms -> 468 ms (1.50x)
deleted=50% (750k records, 250k live): 899 ms -> 549 ms (1.64x)
deleted=90% (950k records, 50k live): 1034 ms -> 578 ms (1.79x)
deleted=99% (995k records, 5k live): 1042 ms -> 577 ms (1.81x)
Same pattern as KUDU-3778: the streaming-path time scales with total
records (creates + deletes), the in-memory-path time only scales with
the live-record decrypt work and a single bulk read per container.
Container-density sweep (500x1000 / 8 dirs / 90% deletion, all 50k
live; steady-state, the first reopen of a fresh on-disk layout is
excluded as it is dominated by cold page-cache):
log_container_max_blocks=-1: 1018 ms -> 577 ms (1.76x)
log_container_max_blocks=10000: 54 ms -> 34 ms (1.59x)
log_container_max_blocks=1000: 28 ms -> 22 ms (1.27x)
Non-encryption regression check (500 x 1000, 8 dirs, 90% deletion,
encryption DISABLED): 805 ms -> 381 ms (2.11x). Matches the numbers
KUDU-3778 reported for the same configuration (811 ms -> 394 ms), so
this change does not regress the existing non-encrypted fast path.
The absolute speedup on encrypted clusters (~1.7x) is lower than on
non-encrypted clusters (~2.1x) because per-record decryption work has
to run on both paths; the in-memory path only collapses the O(records)
preadv() syscalls into a single bulk read per container, it cannot
skip decryption. Both the absolute and relative gain still grow with
the number of records per container, as expected.
Change-Id: I37af5bc37613dd4081e1ad708f084e1c88fb3e75
Reviewed-on: http://gerrit.cloudera.org:8080/24387
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Alexey Serbin <[email protected]>
> Support encrypted metadata in the log block container in-memory replay path
> ---------------------------------------------------------------------------
>
> Key: KUDU-3779
> URL: https://issues.apache.org/jira/browse/KUDU-3779
> Project: Kudu
> Issue Type: Improvement
> Reporter: Daojiang Yan
> Assignee: Daojiang Yan
> Priority: Minor
> Attachments: KUDU-3779-bench-logs.tar.gz
>
>
> KUDU-3778 change introducing
> --log_container_metadata_inmem_replay_threshold_bytes) added an in-memory
> replay fast path for LogBlockContainerNativeMeta::ProcessRecords(): instead
> of issuing two preadv() syscalls per record, the entire metadata file is
> slurped into memory once and ReadablePBContainerFile then reads from a
> MemoryReadableFile wrapper. This drastically reduces syscalls during the
> "Reading filesystem" stage of tserver startup on hosts with many containers.
> The current implementation intentionally falls back to the streaming path
> whenever the metadata file is encrypted (i.e.
> RandomAccessFile::GetEncryptionHeaderSize() > 0). As a result, encrypted
> clusters do not benefit from the optimization.
> *Why encrypted metadata is excluded today*
> ReadablePBContainerFile recovers from an incomplete trailing record (typical
> after a crash mid-write, when the on-disk tail is a file-hole filled with
> zeros) by relying on a per-slice optimization inside DoDecryptV(): it detects
> all-zero ciphertext slices and leaves them as zeros in plaintext rather than
> running them through AES. The PB container parser then sees a zeroed tail and
> returns Status::Incomplete, which the caller handles by truncating the
> partial record.
> If we preload the whole payload as a single buffer and serve all subsequent
> reads from memory, the decryption layer sees one bulk slice instead of
> per-record slices. The trailing zero region is no longer aligned with a slice
> boundary, the zero-slice shortcut is defeated, and the file-hole tail
> decrypts to garbled plaintext. The PB parser would then misclassify a benign
> crash-truncation as corruption.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
