kbendick commented on pull request #3213: URL: https://github.com/apache/iceberg/pull/3213#issuecomment-946038474
> Hi folks, > > I am an Apache Flink contributor and we are currently evaluating different options for this "small-file" problem. So far we only support the compaction of small files when using the StreamingFileSink [1]. Apparently, you have implemented your own way of writing files before committing them to Iceberg I'd also be happy to know the reason. > > We are open to feedback on what you need from Flink to support your use case because we are seeing a lot of other projects hitting the same problems (Hive, DeltaLake etc.) > > [1] https://github.com/apache/flink/blob/ac203a7a26d78fe270bd9941bb721be53a893040/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/stream/StreamingSink.java#L82 Hi @fapaul! I wanted to firstly thank you for dropping by and introduce myself. I believe Ryan already answered your questions, but if you have any other questions you can always feel free to reach out to me (or anyone) and I'd be happy to help or try to point you in the right place. I often work or review Flink-Iceberg related stuff and there are some things I've seen here and there that might have been helpful for us or would be helpful for us in the future. Hope to collaborate with you over there or in here in the future! 🙂 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
