zhongqishang commented on issue #2807: URL: https://github.com/apache/amoro/issues/2807#issuecomment-2095934152
@deniskuzZ Yes, I agree the point. Amoro provides the ability to continuously merge data files and delete files, including position delete file. In Amoro, rewrite position delete file needs to trigger rules. Rewirte data file and position delete file will be performed under the following two conditions. - The number of data files smaller than 16m + eq delete file count exceeds `self-optimizing.minor.trigger.file-count`[1] - The position delete record count associated with a data file exceeding 16m is greater than 10% of the data file record count (self-optimizing.major.trigger.duplicate-ratio)[2] The triggering rule is to reduce the rewrite amplification problem caused by continuous data writing. Maybe a periodic rewrite all file (`self-optimizing.full.rewrite-all-files`)might help you[3]. [1] : https://github.com/apache/amoro/blob/master/amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/optimizing/plan/CommonPartitionEvaluator.java#L321 [2] : https://github.com/apache/amoro/blob/master/amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/optimizing/plan/CommonPartitionEvaluator.java#L198-L214 [3] : https://amoro.apache.org/docs/0.6.1/configurations/#data-cleaning-configurations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
