fapaul commented on pull request #3213: URL: https://github.com/apache/iceberg/pull/3213#issuecomment-934350667
Hi folks, I am an Apache Flink contributor and we are currently evaluating different options for this "small-file" problem. So far we only support the compaction of small files when using the StreamingFileSink [1]. Apparently, you have implemented your own way of writing files before committing them to Iceberg I'd also be happy to know the reason. We are open to feedback on what you need from Flink to support your use case because we are seeing a lot of other projects hitting the same problems (Hive, DeltaLake etc.) [1] https://github.com/apache/flink/blob/ac203a7a26d78fe270bd9941bb721be53a893040/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/stream/StreamingSink.java#L82 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
