pingtimeout commented on issue #2365: URL: https://github.com/apache/polaris/issues/2365#issuecomment-3723184230
As far as I can tell, here is the space complexity for each method. The names used correspond to: * `PM` = number of previous metadata files * `S` = number of snapshots * `ST` = number of statistics files * `PST` = number of partition statistics files * `UM` = number of unique manifest files across all snapshots * `T` = total number of created TaskEntities The `getMetadataFileBatches` method has a space complexity of `O(PM + S + ST + PST)`. Same thing for the `getMetadataTaskStream` method. The `getManifestTaskStream` method has a space complexity of `O(UM)`. The `handleTask` method has a space complexity of `O(UM + PM + S + ST + PST + T)` Based on those elements, it is clear that the current implementation will run into heap pressure for tables with many snapshots and frequent updates, or tables with long metadata history. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
