dragonknightk opened a new pull request, #1394:
URL: https://github.com/apache/paimon-rust/pull/1394

   ### Purpose
   
   Linked issue: close #1230
   
   INSERT INTO a Paimon table through DataFusion encodes every data file in
   one TableWrite, so Parquet encoding uses one core no matter how many input
   partitions DataFusion runs. This change lets PaimonDataSink deal input
   batches to several TableWrites when a bucket-unaware append-only table sets
   `sink.parallelism`, and commits their output as one snapshot.
   
   DataSinkExec still requires a single input partition. DataFusion's own
   DataSink docs leave splitting the input to the sink, and the file sinks do
   the same inside `write_all`, so the change stays inside PaimonDataSink and
   adds no execution plan node. Upstream input partitions already run in
   parallel under the coalesce. Inside one TableWrite the (partition, bucket)
   groups of a batch are written in turn and only rolled files close in
   background tasks, so encoding was the serial part.
   
   The writer count is min(`sink.parallelism`, session `target_partitions`).
   The cap keeps the count within the parallelism the session already grants
   to its operators. Tables shared with Flink often carry `sink.parallelism`
   for Flink jobs; such a table, when it is bucket-unaware append-only, now
   also writes in parallel from DataFusion, with at most `target_partitions`
   writers whatever the Flink value is. The option is validated for every
   table before the sink decides whether it applies, with the forms Java
   Integer.parseInt accepts, so an invalid value fails on any table.
   
   How the writers become one commit: all TableWrites come from one
   WriteBuilder, so they share its commit user and its DataFusion memory
   reservation. Each writer runs in a SpawnedTask that drains a one-slot
   channel and then calls `prepare_commit`. After the input ends, the sink
   joins every task, concatenates their CommitMessages and passes them to a
   single `TableCommit::commit`. Data file names carry a UUID made per file
   writer, so messages from different writers for the same partition never
   collide, and the commit produces one snapshot exactly as it does for one
   writer with several partitions. If any writer or the input fails, the
   statement returns the error before the commit, so no snapshot is written;
   files that other writers already closed stay on storage uncommitted, as
   with a failed single-writer INSERT.
   
   Only bucket-unaware (`bucket = -1`) append-only tables use several writers,
   because there any writer may own any file of a partition. Fixed-bucket
   append tables, primary-key tables, Format Tables and INSERT OVERWRITE keep
   the single writer.
   
   Differences from Java (Flink):
   - `sink.parallelism` is a Flink connector option, and Java's
     FlinkSinkBuilder also reads it from the table options; that is why this
     change reuses the name. Without it Flink derives the writer count from
     the job parallelism; here the absence of the option keeps one writer,
     the current behavior. Flink does not cap the value at the input
     parallelism.
   - Flink also honors the option for fixed-bucket and primary-key tables by
     shuffling rows by bucket, and for overwrite; this change does not.
   - Flink rebalances records; this sink deals whole RecordBatches.
   - `partition.sink-strategy` is not implemented. Each writer writes every
     partition it receives rows for, so on a partitioned table the file count
     multiplies by the writer count. Java's default strategy NONE behaves the
     same way; its HASH strategy, which routes a partition to fewer writers,
     has no counterpart here.
   
   ### Brief change log
   
   - PaimonDataSink validates `sink.parallelism` for every table, and for
     bucket-unaware append-only INSERT INTO uses min(`sink.parallelism`,
     `target_partitions`) TableWrites: input batches are dealt round-robin to
     spawned writers and their combined messages are committed once.
   - The single-writer path is unchanged apart from moving into a branch.
   - docs/src/sql.md documents the option, the cap, the file count on
     partitioned tables and the effect on tables configured for Flink.
   
   ### Tests
   
   New tests in `paimon-datafusion` (table/mod.rs) insert 8 in-memory batches
   of 2 rows into a memory-backed table with `target_partitions = 8` unless
   stated, and read the table back:
   
   - `sink.parallelism = 4`: all 16 rows, one snapshot, and four data files of
     4 rows each (8 batches dealt round-robin to 4 writers). The previous sink
     leaves one 16-row file and fails this test.
   - Partitioned by a column with 2 values, every batch holding one row of
     each: 16 rows, one snapshot, and 4 writers x 2 partitions = 8 files.
   - `sink.parallelism = 4` with `target_partitions = 2`: two 8-row files.
   - Without the option: one 16-row file, the existing behavior.
   - Table-driven, each with `sink.parallelism = 4`: a fixed-bucket table, a
     primary-key table and INSERT OVERWRITE each leave one 16-row file.
     Removing the primary-key guard or the overwrite guard from the sink makes
     this test fail with four 4-row files.
   - Table-driven invalid values: "0", "-1" and " 4" on an append table and
     "0" on a primary-key table fail with a configuration error.
   
   The tests assert the writer count through the file layout. They do not
   assert that the writers run concurrently, and they do not measure CPU
   utilization.
   
   ### API and Format
   
   No public API or file format change. `sink.parallelism` is a new table
   option read by the DataFusion integration. Tables that already carry the
   option, for example from Flink, now get parallel INSERT INTO when they are
   bucket-unaware append-only tables, and fail INSERT if the value is not a
   positive integer.
   
   ### Documentation
   
   docs/src/sql.md, INSERT INTO section.
   
   ## AI Disclosure
   
   Human-led contribution. Grok Bot, working for dragonknightk, assisted with 
this change.
   All submitted content was reviewed by a human before submission.
   
   Assumptions and open points:
   - Reusing the Flink key `sink.parallelism` instead of a DataFusion session
     option is a naming choice that reviewers may want to revisit.
   - The dispatcher waits on the next writer's one-slot queue, so a slow writer
     holds back the others; there is no work stealing between writers.
   - All writers draw on one memory reservation, so a high writer count can
     reach the DataFusion memory limit sooner than one writer would.
   
   (๑•̀ㅂ•́)و✧
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to