raunaqmorarka opened a new pull request, #18076:
URL: https://github.com/apache/iceberg/pull/18076
`ManifestMergeManager.createManifest` reads the manifests of a bin one at a
time
on the thread that writes the merged manifest. On a table that receives many
small appends, the commit that triggers a merge pays one object store read
per
manifest in sequence, so a bin of 100 manifests stalls that commit for
seconds
while ordinary commits take tens of milliseconds.
This change reads the manifests of a bin on the worker pool ahead of the
writer,
which consumes them in bin order. The merged manifest keeps the same entry
order
and the same ADDED/EXISTING/DELETED classification as before.
- Reads run on the existing worker pool, so engines that call
`scanManifestsWith` keep control of the threads used. A bin waiting on a
queued read runs it inline, so a fixed pool with every thread busy on a bin
cannot deadlock.
- Read-ahead is bounded at 1,000 entries per bin, using the entry counts from
the manifest list. Entries handed across threads are full copies with their
per-column stats, which is tens of KB each on a wide table, so the bound
holds
tens of MB per bin at most.
- A manifest larger than the bound, or one with unknown entry counts, is
streamed by the writer as before while reads of the manifests after it run
ahead.
Bin packing, merge thresholds and the merge result cache are unchanged.
Measured with 205 single-file appends on a v2 table, default merge settings,
20 ms added per manifest open to mimic an object store read, and an 8 thread
pool:
| Commit | before | after |
|---|---|---|
| append that merges 100 manifests | 2517 ms | 422 ms |
| ordinary append | ~50 ms | ~50 ms |
Tests: `TestManifestMergeManager` compares the concurrent merge with a
sequential merge over a bin of 100 manifests, covers a manifest above the
bound,
manifests without entry counts, and read failure, and checks that at most a
pool's worth of input streams are open at once (previously 1, now up to the
pool size).
AI assistance (Claude Code) was used to develop this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
