ravjot28 opened a new pull request, #20624:
URL: https://github.com/apache/hudi/pull/20624
### Describe the issue this Pull Request addresses
Closes #20623
In the Java client's commit path, bucket dispatch is serial:
`BaseJavaCommitActionExecutor.execute` writes the buckets
of a commit (each one existing file group to update, or an insert bucket
that may roll over into several new file
groups) one after the other on the calling thread, so a write that touches
many file groups takes the sum of their
write times. Spark writes the same buckets in parallel tasks.
### Summary and Changelog
Java client users can write the file groups of a commit on several threads
with the new
`hoodie.java.write.max.parallelism` (default 1, which keeps the serial
write).
- `HoodieWriteConfig`: new advanced config
`hoodie.java.write.max.parallelism` (since 1.3.0, default 1), getter,
builder method, and a check that the value is positive.
- New `JavaParallelTaskRunner` (`hudi-java-client`): applies a task to a
list on a fixed pool of up to N daemon
threads created for one call (`CustomizedThreadFactory`; the workers use
the caller's context classloader). Workers
take the next item until none is left. Taking an item and recording a stop
(a task failure, or an interrupt of the
caller) hold one lock, so once a stop is recorded no item is taken, while
the items already taken run to their end.
It returns or throws only after the pool has terminated, so no task runs
any more. Results keep the input order. The
first task failure recorded is thrown as is (a checked one wrapped in
`HoodieException`), with the other distinct
failures suppressed; if a pool thread cannot be created, that failure is
thrown instead, with the task failures
suppressed. The caller's interrupt flag is kept on every exit, including
that one. With N = 1 or one item it runs on
the calling thread.
- `BaseJavaCommitActionExecutor.execute`: with the config above 1 and more
than one bucket, the buckets are written
through the runner. Each task drains its bucket's statuses, so insert
buckets (whose files are written while the
lazy iterator is consumed) are written on the pool thread too. With the
default the existing loop runs unchanged.
Why not `HoodieJavaEngineContext.map`: Flink's context honours its
`parallelism` argument with a pool per call, and
Java compaction merges file groups concurrently through list data, so this
fan-out has precedents. But every
Java-engine caller of `context.map` passes a value sized for Spark (the
finalize-write and cleaner parallelisms
default to 200, the rollback parallelism to 100, file listing up to 1,500),
so making the Java context honour it
would not be opt-in. And the write stage needs two guarantees `map` does not
give: no new bucket starts after a
failure, and no bucket task is still running when the exception leaves
`execute`.
Why not `ExecutorServiceBasedEngineContext`
(https://github.com/apache/hudi/pull/18585): its `map` runs on one
JVM-wide pool sized to the common pool's parallelism and ignores the
`parallelism` argument, so a user cannot bound
the file groups in flight per write; it submits every item up front and
waits for all of them, so after one bucket
fails the remaining buckets are still written before the exception arrives;
and its workers carry the classloader
that loaded Hudi rather than the caller's.
Why a new key: `hoodie.upsert.shuffle.parallelism`,
`hoodie.insert.shuffle.parallelism` and
`hoodie.bloom.index.parallelism` describe Spark shuffle partitions and are
ignored by the Java engine today; reusing
them would give existing Java users that many threads on upgrade, with no
opt-in. The name follows the engine-scoped
`hoodie.spark.*` keys and the `max.parallelism` keys that bound concurrent
work (`hoodie.clustering.max.parallelism`),
and it is named for the write as a whole: a follow-up applies the same bound
to the BLOOM index key check that tags
the records of a write, so one value bounds the file groups a Java write
works on at once.
Thread safety follows the Java compaction path, which already writes file
groups concurrently with one shared
`HoodieTable`: every bucket has its own handle and write status; mergers are
created per handle; the partitioner and
the file system view are only read during the stage (the view manager getter
is synchronized and views come from a
concurrent map); file names stay distinct by file id under the constant Java
write token; markers are one per file;
the partition metafile is written through a temporary name. Lifecycle calls
(heartbeat, commit) stay outside the
stage. The metadata table's write config does not inherit the key, so its
bucket-write stage keeps parallelism 1.
### Impact
No breaking API change. Additive: the config
`hoodie.java.write.max.parallelism` with
`HoodieWriteConfig#getJavaWriteMaxParallelism` and
`HoodieWriteConfig.Builder#withJavaWriteMaxParallelism`, and the
public utility class `org.apache.hudi.client.common.JavaParallelTaskRunner`
in `hudi-java-client`. With the default
nothing changes. With a value above 1, peak memory of a write grows with the
number of file groups in flight (each
holds its merge map, up to `hoodie.memory.merge.max.size`, the row group it
is building, on merge-on-read tables the
log block it is buffering, and with a queue-based write executor its queue
and threads); the config doc says so.
When a bucket fails, the stage throws only after every bucket task it
started has returned. Within a bucket task,
cleanup is the serial path's: with a queue-based write executor,
`JavaLazyInsertIterable.computeNext` calls
`shutdownNow()` and then `awaitTermination()`, which waits up to 60 seconds
for the producer threads and then up to 60
seconds for the consumer thread; the iterable does not check whether they
ended, and if `shutdownNow()` throws, the
executors are not shut down and nothing is awaited. This PR does not change
that code, so it does not guarantee that
those nested threads have ended when the exception reaches the caller. The
failure tests below cover a merge failure
and a queue-based insert failure, where they have.
Local timing, to show the effect of the setting on one machine (not a
benchmark): a shared workstation (AMD Ryzen 5
7600, 6 cores), the build container limited to 4 CPUs (`docker --cpus=4`),
local NVMe disk, JDK 11 with the module's
test JVM settings (`-Xmx4g`, parallel GC). Each run takes a fresh copy of a
copy-on-write table of 400,000 rows in 16
partitions (file groups of up to 6,250 rows, metadata table off) and writes
the same 160,000 records to it with
`upsertPreppedRecords`, then commits: 80,000 updates spread over all
existing file groups and 80,000 new records, one
new file group per partition, so 80 or more file groups per write. Wall time
covers the write and the commit. The four
values ran in random order, one warm-up each, then 7 measured runs each.
| `hoodie.java.write.max.parallelism` | Wall time, median (fastest -
slowest) | Rows per second (median) | Speed-up | Peak heap after GC (median) |
|---|---|---|---|---|
| 1 | 7.30 s (7.21 - 7.49) | 21,924 | 1.00 | 683 MB |
| 2 | 4.01 s (3.96 - 4.39) | 39,930 | 1.82 | 691 MB |
| 4 | 2.30 s (2.23 - 2.45) | 69,595 | 3.17 | 708 MB |
| 8 | 2.78 s (2.67 - 2.99) | 57,492 | 2.62 | 744 MB |
Rows per second is 160,000 divided by the median wall time; speed-up is the
median at 1 divided by the median at each
value. Peak heap is the largest heap in use right after a garbage collection
during the write, read from the
collectors' after-collection figures every 10 ms; before each write it was
575 MB (input records and test state), so
the write added about 108 MB at 1 thread and 169 MB at 8.
What these numbers show: with 4 CPUs a write took 55 % of the serial time at
2 threads and 32 % at 4; at 8, above
the CPU count, it was slower than at 4, so here a value above the available
cores did not help. The file groups here
are small, so the heap figures do not show the cost for large file groups,
where the per-file-group terms in the
config doc apply. An earlier run of the same measurement on an earlier
version of this change gave the same picture
(medians 7.22, 3.88, 2.19 and 2.71 s).
### Risk Level
low. Opt-in; with the default the existing serial loop runs. With the option
on, the tests below show the table, its
files, commit metadata and timeline equal to the serial write, and a failing
file group stops the write with the
serial exception after the other bucket tasks have finished.
Tests, run on master with JDK 11 and checkstyle on for main and test sources:
- `TestJavaParallelTaskRunner` (16): order with out-of-order completion,
empty and single input, at most N tasks at
once, one thread per item below N; failure: the item running when the
failure is recorded finishes before the call
returns and no further item starts, and a thread held just before it takes
an item until another thread's failure
is recorded takes none (both coordinated on the recorded stop, no sleeps);
one primary and the other failures
suppressed; the same exception from two tasks thrown unchanged; errors
unwrapped; checked failures wrapped; caller
interrupt (waits for the running items, keeps the flag); a thread that
cannot be created (the running item finishes
first, the caller's interrupt flag is kept, the submission failure is
thrown), and the same after a task failure
(the submission failure is thrown with the task failure suppressed);
context classloader; no thread left.
- `TestHoodieJavaClientConcurrentWrites` (30):
- concurrency measured inside the write (a test write status holds the
first record of each file group until 4
file groups are writing) for insert buckets, update buckets,
merge-on-read log appends and a queue-based write
executor;
- parity against the serial write, where each step runs on a table at
parallelism 1 and on a byte copy of it at
parallelism 4 (so both start from the same file groups) and compares
rows, file groups by record keys, base and log
files per file group, commit metadata counts, completed instants and the
partition metafile. The full sequence
(insert, upsert, `insertPreppedRecords`, delete, bulk insert, plus
`upsertPreppedRecords` and insert overwrite on
copy-on-write) runs for copy-on-write and merge-on-read with the
embedded timeline server on and off, and once with
`BOUNDED_IN_MEMORY`; a smaller copy-on-write insert and upsert is
repeated 10 times; and a rollover inside a bucket;
- record errors counted as in the serial write;
- failures: a corrupt base file failing an upsert, with direct and
timeline-server markers, while a sibling file
group is held until the failing file group's writer has started and the
stage's other pool threads have exited,
which they do only after the failure is recorded: same exception as the
serial write, the sibling's file is
complete when the call throws, no write thread is alive, every created
file has a marker, and rollback restores
the table; and an insert that fails inside a queue-based write executor,
with the same sibling, thread, marker and
rollback checks, plus no queue thread of the write left;
- default and single-bucket writes stay on the calling thread; the
metadata write config keeps 1; values below 1
are refused.
- Without the main code change (`BaseJavaCommitActionExecutor` as on master,
everything else kept), 9 of the 30 fail
in every rerun: the 6 concurrency tests (the gate times out, as file
groups are written one at a time) and the 3
failure tests (the failing file group cannot run while the sibling is
still being written); with it all pass.
- The claim test fails against a variant of the runner that checks the stop
flag and then takes the next index in
two steps (the held thread has passed the check, so it takes an item after
the failure is recorded); with the one
lock it passes.
- Existing suites with the default pass:
`TestJavaCopyOnWriteActionExecutor`, `TestHoodieJavaClientOnCopyOnWriteStorage`,
`TestHoodieJavaClientOnCopyOnWriteStorageForTestFormat`,
`TestHoodieJavaClientOnMergeOnReadStorage`,
`TestJavaHoodieBackedMetadata`, `TestHoodieWriteConfig`.
### Documentation Update
The new config is documented in its `withDocumentation` text; the
configurations page is generated from it.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]