leaves12138 commented on code in PR #1027:
URL: https://github.com/apache/paimon-rust/pull/1027#discussion_r4177379662
##########
crates/paimon/src/arrow/format/mod.rs:
##########
@@ -470,111 +470,179 @@ fn supported_write_formats() -> Vec<&'static str> {
]
}
-/// Create a format writer that streams directly to storage.
+/// Reusable factory for one rolling writer, mirroring Java
FormatWriterFactory.
+/// Output and compression belong to each file; schema/options belong to the
factory.
+#[async_trait]
+pub(crate) trait FormatWriterFactory: Send + Sync {
+ async fn create_writer(
+ &self,
+ output: &OutputFile,
+ compression: &str,
+ ) -> crate::Result<Box<dyn FormatFileWriter>>;
+
+ /// Some plan factories need the previous close callback before the next
file.
+ fn needs_completed_file_stats(&self) -> bool {
+ false
+ }
+}
+
+/// Build a factory once per rolling writer. Statistics describe logical value
+/// fields independently of a shredding plan's physical schema (notably for PK
files).
+pub(crate) fn create_format_writer_factory(
+ file_format: &str,
+ schema: SchemaRef,
+ zstd_level: i32,
+ file_io: Option<FileIO>,
+ write_fields: Option<&[DataField]>,
+ format_options: Option<&HashMap<String, String>>,
+ stats_fields: Option<&[DataField]>,
+) -> crate::Result<Arc<dyn FormatWriterFactory>> {
+ let file_format = file_format.rsplit('.').next().unwrap_or_default();
+ if file_format.eq_ignore_ascii_case("parquet") {
Review Comment:
Thanks, this is addressed: unsupported ORC MAP shared-shredding is now
rejected before writing, with coverage for schema validation, direct writers,
and append/PK writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
