JingsongLi commented on code in PR #1394:
URL: https://github.com/apache/paimon-rust/pull/1394#discussion_r4581460134


##########
crates/integrations/datafusion/src/physical_plan/sink.rs:
##########
@@ -116,6 +126,82 @@ impl PaimonDataSink {
             overwrite,
         }
     }
+
+    /// Number of `TableWrite`s for this statement, capped by the session's
+    /// `target_partitions`. The option is validated for every table. Any
+    /// writer may write any partition only for bucket-unaware append-only
+    /// tables, so other tables and overwrites keep the single writer.
+    fn writer_count(&self, target_partitions: usize) -> DFResult<usize> {
+        let options = self.table.schema().options();
+        let parallelism = match options.get(SINK_PARALLELISM) {
+            None => return Ok(1),
+            // Same accepted forms as Java Integer.parseInt.
+            Some(value) => match value.parse::<i32>() {
+                Ok(count) if count > 0 => count as usize,
+                _ => {
+                    return Err(DataFusionError::Configuration(format!(
+                        "'{SINK_PARALLELISM}' must be a positive integer, got 
'{value}'"
+                    )))
+                }
+            },
+        };
+        let core_options = CoreOptions::new(options);
+        if self.overwrite
+            || core_options.is_format_table()
+            || !self.table.schema().primary_keys().is_empty()
+            || core_options.bucket() != -1
+        {
+            return Ok(1);
+        }
+        Ok(parallelism.min(target_partitions.max(1)))
+    }
+
+    /// Deal input batches round-robin to `writers` spawned `TableWrite`s and
+    /// gather their commit messages.
+    async fn write_parallel(
+        &self,
+        wb: &WriteBuilder<'_>,
+        writers: usize,
+        mut data: SendableRecordBatchStream,
+    ) -> DFResult<(u64, Vec<CommitMessage>)> {
+        let mut senders = Vec::with_capacity(writers);
+        let mut tasks = Vec::with_capacity(writers);
+        for _ in 0..writers {
+            let mut tw = wb.new_write().map_err(to_datafusion_error)?;
+            // One queued batch per writer bounds buffered input.
+            let (sender, mut receiver) = mpsc::channel::<RecordBatch>(1);
+            senders.push(sender);
+            tasks.push(SpawnedTask::spawn(async move {
+                while let Some(batch) = receiver.next().await {
+                    tw.write_arrow_batch(&batch).await?;
+                }
+                tw.prepare_commit().await
+            }));
+        }
+
+        let mut row_count = 0u64;
+        let mut next = 0;
+        while let Some(batch) = data.next().await {

Review Comment:
   [P2] Observe writer failures while waiting for upstream input
   
   A writer's error is only collected after EOF or after another send reaches 
its disconnected channel. If a writer fails and the source is awaiting its next 
batch, this `data.next().await` never observes that failure; the INSERT can 
stay blocked indefinitely instead of reporting the storage error and canceling 
the source. Joining tasks in creation order after EOF can similarly hide a 
failed later writer behind a stalled earlier one.
   
   I reproduced the primary case through PaimonDataSink::write_all with a real 
filesystem NotADirectory failure, write-buffer-size=1b, and a source that 
yields one valid batch then remains Pending (modeling a slow remote source). 
One writer returns the I/O error immediately; two writers hit the 500ms test 
timeout, and remain blocked because the source never finishes. Poll input/sends 
and writer completions together, propagate the first failure and cancel the 
remaining work. Add a failed-writer plus pending-input regression.



##########
crates/integrations/datafusion/src/physical_plan/sink.rs:
##########
@@ -116,6 +126,82 @@ impl PaimonDataSink {
             overwrite,
         }
     }
+
+    /// Number of `TableWrite`s for this statement, capped by the session's
+    /// `target_partitions`. The option is validated for every table. Any
+    /// writer may write any partition only for bucket-unaware append-only
+    /// tables, so other tables and overwrites keep the single writer.
+    fn writer_count(&self, target_partitions: usize) -> DFResult<usize> {
+        let options = self.table.schema().options();
+        let parallelism = match options.get(SINK_PARALLELISM) {
+            None => return Ok(1),
+            // Same accepted forms as Java Integer.parseInt.
+            Some(value) => match value.parse::<i32>() {
+                Ok(count) if count > 0 => count as usize,

Review Comment:
   [P2] Preserve the valid Flink default-parallelism sentinel
   
   `sink.parallelism=-1` is a legal existing Flink table configuration meaning 
inherit default parallelism, not an invalid count. Paimon's 
[FlinkSinkBuilder](https://github.com/apache/paimon/blob/f289b0a2829f8b0c28ae92743f11893893ce1116/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java#L471)
 explicitly treats null or -1 as undefined; it is passed through to Flink's 
supported PARALLELISM_DEFAULT. Such a shared table was writable by DataFusion 
before this PR but now every INSERT fails here, even for 
PK/fixed-bucket/overwrite paths where the option is not used.
   
   I reproduced an ordinary SQL INSERT with the existing EightBatchInsert 
public write/read harness and only `sink.parallelism=-1`: it returns 
Configuration("...must be a positive integer...") before writing. Handle the 
inheritance sentinel consistently (retaining the prior single-writer behavior 
is a safe default), keep rejecting truly invalid values, and add shared-table 
compatibility coverage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to