baiyangtx commented on PR #7866: URL: https://github.com/apache/paimon/pull/7866#issuecomment-4515909075
> Hi @baiyangtx @Aitozi , please take a look to #7842 , I didn't look closely, maybe the abilities of these two PR implementations are the same. Thanks @umi for working on this too. My PR #7866 takes a different approach — sorting inline during the existing full compaction path, rather than a separate post-commit sort pipeline. Key differences: - **No new compact type** — sorting happens inside `ManifestFileMerger.tryFullCompaction`, reusing the existing full compaction trigger and lifecycle. No additional manifest scans or rewrite passes. - **Memory-safe with external sort** — uses `BinaryExternalSortBuffer` (already battle-tested across Paimon). When entries exceed `manifest.merge.sort.buffer`, it automatically spills to disk. No risk of OOM regardless of manifest count. - **Finer-grained config** — separate toggles for merge compaction, commit-time merge, and delta writing, instead of a single switch. Happy to align with the community's direction, but I believe the inline approach is simpler and less invasive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
