steFaiz commented on PR #9177: URL: https://github.com/apache/paimon/pull/9177#issuecomment-5262273540
Thanks for this PR! I totally agree with that! Just one concern: I'm thinking about the two-phase plan: * In AI workload, DataEvolution tables will be frequently added new columns. I think most compactions will be caused by column-level small files. DE tables are all batch-oriented, so row-level small files are more scarce. * For column-level compaction, almost all files will be added as candidates. Will this two-phase plan reduce memory usage at that case? If not, we actually scan manifest twice but save no memory. * But I think this is useful for Materialize-Compaction job -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
