prashantwason opened a new issue, #20673:
URL: https://github.com/apache/hudi/issues/20673
### Task Description
**What needs to be done:**
Make record index (and other metadata table partition) compaction cheaper. A
metadata table file group compaction reads the base HFile and its log files and
rewrites the whole file group. On large record index file groups (tens of
millions of entries), almost all of the time goes to rewriting entries that no
log record touches.
We profiled compaction (JFR, exclusive attribution) with a standalone
benchmark that runs the real compaction code on two production record index
file groups: Table A has 34M entries and a 1.9 GB base file; Table B has 105M
entries and an 8.3 GB base file.
| Component | Table A | Table B |
|---|---|---|
| Avro decode | 25.9% | 20.8% |
| Avro encode | 21.8% | 16.7% |
| gzip compression | 19.4% | 21.3% |
| Bloom filter add | 16.6% | 19.9% |
| Waiting on storage reads | 7.3% | 10.6% |
| HFile internals | 3.5% | 5.3% |
| Record key extraction | 2.1% | — |
| Merge (file group reader, record buffer, wrappers) | 1–2% | 1–2% |
| gzip decompression | 1.5% | 2.2% |
The merge itself is almost free: more than 99.9% of base entries have no log
record for their key. Removing the bloom filter is handled in #20666, which
measured a 5–10% saving. That saving is lower than the profile share, so treat
the estimates below as upper bounds until each is benchmarked.
Proposed improvements:
**1. Copy untouched entries as raw bytes (estimate 40–50%)**
Every base value is decoded with `GenericDatumReader` and a
`ResolvingDecoder`, using a schema that adds the `_hoodie_*` meta fields, and
wrapped in a `HoodieRecord`. `HoodieAvroHFileWriter.writeAvro` then re-encodes
it with `HoodieAvroUtils.avroToBytes`. Instead, a record-index merge handle
could:
- copy the stored bytes for keys with no log record, using
`HFileWriterImpl.append(String, byte[])`, since the reader already holds the
value bytes;
- decode only the keys that have a log record, and merge them through
`HoodieMetadataPayload`.
Constraints:
- The output file must keep the input file's Avro schema, and merged records
must be encoded with it.
- Deletes are not a problem: they never reach base files, and log deletes
only hit keys that are decoded anyway.
- Gate it behind a config, initially for the record index partition only.
**2. Cheaper Avro decode and encode for entries that must be materialized
(estimate 10–15% without #1)**
- Reuse the `BinaryEncoder` and output buffer in the HFile writer instead of
creating new ones per record.
- Decode with the file's own schema instead of resolving to a schema with
the meta fields added.
- Avoid the per-record cost of union resolution: about 8% on Table A goes to
union branch lookup, which hashes a String for every union field of every
record.
#1 makes most of this unnecessary for compaction, but it still helps
log-block writes and the keys that do get merged.
**3. Cheaper HFile block compression (up to ~20%)**
`HoodieAirliftGzipCompressor` opens a new gzip stream for every 1 MB block.
Options to benchmark:
- Reuse the `Deflater` across blocks, and allow a lower gzip level. Both
keep the file format unchanged.
- Add zstd. `CompressionCodec.ZSTD` (id 6) exists, but
`HoodieCompressorFactory.getCompressor` supports only `NONE` and `GZIP`, so
`hoodie.hfile.compression.algorithm=ZSTD` fails today. aircompressor, already a
dependency, ships a pure-Java `ZstdHadoopStreams` that mirrors the gzip
implementation; zstd-jni is the faster native option.
- zstd should decompress much faster, which also helps metadata table
lookups.
- Every reader must support zstd before any writer uses it, because older
readers cannot open such files.
- The metadata table builds its own storage config, so it would need a way
to set the codec.
- Copy whole compressed blocks that no log key touches. Measure how many
blocks a typical compaction touches first: log keys spread across the key space
may touch most blocks.
After #1 and the bloom filter removal, compression is the largest remaining
CPU cost, at about 40% of the remaining time.
**4. Overlap storage reads with CPU (estimate 5–10% of wall time)**
The compaction task waits on base file reads for 7–11% of its time.
Prefetching the next blocks on a separate thread, or reading in larger chunks,
would hide most of it. This cuts wall time, not CPU.
**Why this task is needed:**
Record index compaction is CPU-bound, and its cost grows with the size of
the table rather than the size of the update: a few thousand changed keys force
a rewrite of tens of millions of entries. Combined with #20666, #1 and #3
should make record index compaction about 2× cheaper, which cuts executor time
for compaction and the executors needed to meet a compaction deadline by the
same factor.
### Task Type
Performance optimization
### Related Issues
**Parent feature issue:**
**Related issues:** #20666 (skip writing metadata table HFile bloom filters
unless bloom filter lookups are enabled), #18936
(`hoodie.hfile.bloom.filter.enabled`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]