polyzos commented on issue #3252: URL: https://github.com/apache/paimon/issues/3252#issuecomment-2100002330
**More context on this:** turns out it was the effect of not having compaction running in the background., which wasn't clear in the first place. This means that although partition/snapshot/manifest file expiration happens automatically https://paimon.apache.org/docs/master/maintenance/manage-snapshots/#expire-snapshots it results in more files being generated, but not physically deleted because compaction is disabled. This means that although the generated files keep increasing, they are also never removed, but Paimon will still have to access/list more metadata files (more and more over time), in order to find out what's still relevant and what's not. Turning compaction, in the long run will keep the requests stable. At the same time it is also recommended to use **version 0.8+** that brings many improvements as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
