zhongqishang commented on issue #2807: URL: https://github.com/apache/amoro/issues/2807#issuecomment-2109456088
@ayushtkn Great to hear that you have been exploring Amoro to work with Hive-4.0. If I understand correctly, after optimizer compaction is executed, you expected is to keep only the data files (no pos delete files), right? I think this is a reasonable demand scenario and best read performance can also be achieved. But if the iceberg table with flink stream writer, you need to keep rewriting the data file, which will cause large IO consumption. BTW, only add the count for posDeleteFiles trigger will not be rewrite data file and eq delete file. In the current implementation, only `self-optimizing.major.trigger.duplicate-ratio` can generate Major task(rewrite pos delete file). If possible, could you try setting the `self-optimizing.major.trigger.duplicate-ratio` property of the table to 0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
