JingsongLi commented on code in PR #1394:
URL: https://github.com/apache/paimon-rust/pull/1394#discussion_r4581460134
##########
crates/integrations/datafusion/src/physical_plan/sink.rs:
##########
@@ -116,6 +126,82 @@ impl PaimonDataSink {
overwrite,
}
}
+
+ /// Number of `TableWrite`s for this statement, capped by the session's
+ /// `target_partitions`. The option is validated for every table. Any
+ /// writer may write any partition only for bucket-unaware append-only
+ /// tables, so other tables and overwrites keep the single writer.
+ fn writer_count(&self, target_partitions: usize) -> DFResult<usize> {
+ let options = self.table.schema().options();
+ let parallelism = match options.get(SINK_PARALLELISM) {
+ None => return Ok(1),
+ // Same accepted forms as Java Integer.parseInt.
+ Some(value) => match value.parse::<i32>() {
+ Ok(count) if count > 0 => count as usize,
+ _ => {
+ return Err(DataFusionError::Configuration(format!(
+ "'{SINK_PARALLELISM}' must be a positive integer, got
'{value}'"
+ )))
+ }
+ },
+ };
+ let core_options = CoreOptions::new(options);
+ if self.overwrite
+ || core_options.is_format_table()
+ || !self.table.schema().primary_keys().is_empty()
+ || core_options.bucket() != -1
+ {
+ return Ok(1);
+ }
+ Ok(parallelism.min(target_partitions.max(1)))
+ }
+
+ /// Deal input batches round-robin to `writers` spawned `TableWrite`s and
+ /// gather their commit messages.
+ async fn write_parallel(
+ &self,
+ wb: &WriteBuilder<'_>,
+ writers: usize,
+ mut data: SendableRecordBatchStream,
+ ) -> DFResult<(u64, Vec<CommitMessage>)> {
+ let mut senders = Vec::with_capacity(writers);
+ let mut tasks = Vec::with_capacity(writers);
+ for _ in 0..writers {
+ let mut tw = wb.new_write().map_err(to_datafusion_error)?;
+ // One queued batch per writer bounds buffered input.
+ let (sender, mut receiver) = mpsc::channel::<RecordBatch>(1);
+ senders.push(sender);
+ tasks.push(SpawnedTask::spawn(async move {
+ while let Some(batch) = receiver.next().await {
+ tw.write_arrow_batch(&batch).await?;
+ }
+ tw.prepare_commit().await
+ }));
+ }
+
+ let mut row_count = 0u64;
+ let mut next = 0;
+ while let Some(batch) = data.next().await {
Review Comment:
[P2] Observe writer failures while waiting for upstream input
A writer's error is only collected after EOF or after another send reaches
its disconnected channel. If a writer fails and the source is awaiting its next
batch, this `data.next().await` never observes that failure; the INSERT can
stay blocked indefinitely instead of reporting the storage error and canceling
the source. Joining tasks in creation order after EOF can similarly hide a
failed later writer behind a stalled earlier one.
I reproduced the primary case through PaimonDataSink::write_all with a real
filesystem NotADirectory failure, write-buffer-size=1b, and a source that
yields one valid batch then remains Pending (modeling a slow remote source).
One writer returns the I/O error immediately; two writers hit the 500ms test
timeout, and remain blocked because the source never finishes. Poll input/sends
and writer completions together, propagate the first failure and cancel the
remaining work. Add a failed-writer plus pending-input regression.
##########
crates/integrations/datafusion/src/physical_plan/sink.rs:
##########
@@ -116,6 +126,82 @@ impl PaimonDataSink {
overwrite,
}
}
+
+ /// Number of `TableWrite`s for this statement, capped by the session's
+ /// `target_partitions`. The option is validated for every table. Any
+ /// writer may write any partition only for bucket-unaware append-only
+ /// tables, so other tables and overwrites keep the single writer.
+ fn writer_count(&self, target_partitions: usize) -> DFResult<usize> {
+ let options = self.table.schema().options();
+ let parallelism = match options.get(SINK_PARALLELISM) {
+ None => return Ok(1),
+ // Same accepted forms as Java Integer.parseInt.
+ Some(value) => match value.parse::<i32>() {
+ Ok(count) if count > 0 => count as usize,
Review Comment:
[P2] Preserve the valid Flink default-parallelism sentinel
`sink.parallelism=-1` is a legal existing Flink table configuration meaning
inherit default parallelism, not an invalid count. Paimon's
[FlinkSinkBuilder](https://github.com/apache/paimon/blob/f289b0a2829f8b0c28ae92743f11893893ce1116/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java#L471)
explicitly treats null or -1 as undefined; it is passed through to Flink's
supported PARALLELISM_DEFAULT. Such a shared table was writable by DataFusion
before this PR but now every INSERT fails here, even for
PK/fixed-bucket/overwrite paths where the option is not used.
I reproduced an ordinary SQL INSERT with the existing EightBatchInsert
public write/read harness and only `sink.parallelism=-1`: it returns
Configuration("...must be a positive integer...") before writing. Handle the
inheritance sentinel consistently (retaining the prior single-writer behavior
is a safe default), keep rejecting truly invalid values, and add shared-table
compatibility coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]