JanKaul commented on issue #34: URL: https://github.com/apache/iceberg-rust/issues/34#issuecomment-1685208560
Thanks @Xuanwo for the write up. For me it would be Okay to just start with `MultipartUpload` as it is the most general option and works with all memory requirements and file sizes. We could then later add the optimization to use `PutObject` for small files. And I strongly agree with your point on using the same interface for the parquet and the avro writer. Even when it means we need to implement an async writer with an internal buffer for avro. Looking at the [parquet asyncwriter implementation](https://docs.rs/parquet/45.0.0/src/parquet/arrow/async_writer/mod.rs.html#71-83), it doesn't look too bad. @ZENOTME the interface looks great to me. Thanks for the effort. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
