mikedias commented on PR #7865: URL: https://github.com/apache/paimon/pull/7865#issuecomment-4758728624
@JingsongLi yes, it does apply to fixed buckets. However, for the historical tables and assignments that haven't had partitions rescaled independently, the logic remains the same. Assuming I've covered all scenarios here, historical tables will remain compatible. (FWIW, this patch is working well in our environments so far, and we haven't seen any issues) Regarding Postpone buckets, we have explored using them, but we faced a few challenges: - New data isn't visible before compaction, significantly reducing the freshness of these tables. Compacting at the cadence we need would be prohibitively expensive. - When dealing with thousands of partitions, the Postpone Bucket compaction dag creates too many splits and crashes the Job Manager. Hence, I'm unsure whether Postpone is the best way to handle heavily skewed, highly partitioned tables that can't wait for compaction to show data. I understand the concern around changing the code for the fixed bucket assignments. Is there a way we can mitigate the risk here by breaking this PR into smaller ones or by adding more tests to areas of concern? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
