yandrey321 opened a new pull request, #10709: URL: https://github.com/apache/ozone/pull/10709
## What changes were proposed in this pull request? When computing a file checksum for an erasure-coded key, ECFileChecksumHelper builds a temporary STANDALONE pipeline (replica index 1 + parity nodes) for every block. All blocks of a file share the same EC placement group, so that pipeline is identical for every block — rebuilding it per block (node filtering, deterministic-ID hashing, allocation) is wasted work that grows with the number of blocks. This PR caches the standalone pipeline per placement group inside the helper, so it is built once per file (on the first block) and reused for the rest. It builds on HDDS-15643 (#10594) — already on master — which removed the redundant OM lookupKey RPC and made the pipeline ID deterministic for cross-file connection reuse. This adds the intra-file reuse on top. It is purely client-side CPU savings: no wire, OM/SCM, or checksum-result changes. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15680 ## How was this patch tested? New unit tests in TestFileChecksumHelper cover the cache hit/miss paths, the node filtering, and the acquire-failure path. Benchmarked with FileChecksumBenchmark#runBlockCacheBenchmark (5 threads, each checksumming random-placement EC files), comparing the cache off vs on while holding everything else equal. Results — throughput gain (cache on ÷ off), by file size in blocks EC | Blocks/file | Before | After | Speedup -- | -- | -- | -- | -- RS-3-2 | 1 | 244,351 | 239,566 | ~1.00× RS-3-2 | 3 | 126,262 | 130,494 | 1.03× RS-3-2 | 5 | 86,606 | 91,613 | 1.06× RS-3-2 | 10 | 48,442 | 51,364 | 1.06× RS-6-3 | 1 | 236,776 | 229,178 | ~1.00× RS-6-3 | 3 | 123,013 | 128,283 | 1.04× RS-6-3 | 5 | 84,271 | 88,997 | 1.06× RS-6-3 | 10 | 47,155 | 50,820 | 1.08× Single-block files show no change (nothing to cache — the 1-block row is a control and sits within ~2-3% noise). The gain grows with file size as the cache eliminates the redundant per-block pipeline rebuilds — up to ~6–8% for large multi-block EC files, slightly higher for wider EC schemes (RS-6-3). The improvement is client-side CPU only and does not change the checksum result. As a summary the fix would have slight performance penalty for small files, and single digit improvement for files spans to multiple blocks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
