kumarUjjawal commented on code in PR #19931:
URL: https://github.com/apache/datafusion/pull/19931#discussion_r2726467718
##########
datafusion/core/src/physical_planner.rs:
##########
@@ -549,8 +549,30 @@ impl DefaultPhysicalPlanner {
}
};
+ // Parse single_file_output option if explicitly set
+ let file_output_mode = match source_option_tuples
+ .get("single_file_output")
+ .map(|v| v.trim())
+ {
+ None => FileOutputMode::Automatic,
+ Some("true") => FileOutputMode::SingleFile,
+ Some("false") => FileOutputMode::Directory,
+ Some(value) => {
+ return Err(DataFusionError::Configuration(format!(
+ "provided value for 'single_file_output' was not
recognized: \"{value}\""
+ )));
+ }
+ };
+
+ // Filter out sink-related options that are not format options
+ let format_options: HashMap<String, String> =
source_option_tuples
Review Comment:
The `single_file_output` option is a sink-level configuration that controls
file output behavior, not a format-specific option . The
`file_type_to_format().create(session_state, &format_options)` call validates
options against the format's config schema and would fail with "unrecognized
option" if we pass `single_file_output`. So we extract it first for
`FileSinkConfig`, then filter it out before passing the remaining options to
the format factory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
