ravjot28 opened a new issue, #20623: URL: https://github.com/apache/hudi/issues/20623
### Feature Description **What the feature achieves:** An opt-in setting for the standalone Java write client, `hoodie.java.write.max.parallelism` (default 1), that writes the file groups of one commit on up to that many threads instead of one after the other. **Why this feature is needed:** In the Java client's commit path, bucket dispatch is serial. `BaseJavaCommitActionExecutor.execute` (`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java:115-124` on master a65a573a49f) groups the records into buckets, each one existing file group to update or an insert bucket that may roll over into several new file groups, and writes them in `partitionedRecords.forEach(...)` on the calling thread. A write that touches many partitions or file groups therefore takes the sum of all per-bucket write times. Spark writes the same buckets in parallel tasks (`BaseSparkCommitActionExecutor`, `mapPartitionsWithIndex`), and Flink spreads them over write subtasks. Other parts of the Java engine already run concurrently: list data and `HoodieJavaEngineContext.map` use parallel streams, and Java compaction merges file groups concurrently through list data with one shared `HoodieTable` (`HoodieCompactor.java:142-155`). The regular write is the path left serial. Related: https://github.com/apache/hudi/issues/6585, where raising `withParallelism` and the other parallelism settings did not change the Java client's write time. On the Java engine those values do not set how many file groups are written at once. ### User Experience **How users will use this feature:** - Configuration: set `hoodie.java.write.max.parallelism` to the number of file groups to write at the same time. The default 1 keeps today's serial write. Values below 1 are refused when the write config is built. - API: no breaking change. Additive only: the config with `HoodieWriteConfig#getJavaWriteMaxParallelism` and `HoodieWriteConfig.Builder#withJavaWriteMaxParallelism`, and a small runner class in `hudi-java-client`. - Example: `HoodieWriteConfig.newBuilder()...withJavaWriteMaxParallelism(4).build()`. Behaviour with a value above 1: - The buckets of a commit are written on a fixed pool of up to that many threads, created for the write stage and shut down when the stage ends. Each bucket is written completely on its pool thread, including the insert buckets, whose files are written while their statuses are drained. - Write statuses come back in the same order as with one thread, so commit metadata and the index update see the same input. - When a bucket fails, no further bucket starts (taking the next bucket and recording the failure are atomic); the buckets already taken run to their end, and only then does the stage throw the same exception as the serial write, with any other failures attached as suppressed. If a pool thread cannot be created, that failure is thrown instead, with the bucket failures attached as suppressed. Every bucket task the stage started has returned when the exception reaches the caller. Threads that a bucket's writer starts itself (the producer and consumer of a queue-based write executor) are stopped and awaited by the existing per-bucket code, with the same limits as on the serial path. - The value bounds the file groups in flight, not the threads of the process (a queue-based `hoodie.write.executor.type` adds its own producer and consumer threads per file group being written). - Memory grows with the value: each file group being written holds its own merge map (up to `hoodie.memory.merge.max.size`), the row group it is building, on merge-on-read tables the log block it is buffering, and with a queue-based write executor its queue. - Shuffle parallelism settings such as `hoodie.upsert.shuffle.parallelism` are not reused: their values are sized for Spark (200 is common) and the Java engine ignores them today, so reusing them would change the threads of existing Java users without an opt-in. - The key is named for the write as a whole: a follow-up applies the same bound to the BLOOM index key check that tags the records of a write. ### Hudi RFC Requirements **RFC PR link:** not applicable **Why RFC is/isn't needed:** - Does this change public interfaces/APIs? No breaking change; one new advanced config (with its getter and builder method) whose default keeps today's behaviour, and a new utility class. - Does this change storage format? No. - Justification: the change is local to the Java engine's write stage; files, timeline and metadata are the same as with the serial write. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
