szehon-ho commented on PR #4596: URL: https://github.com/apache/iceberg/pull/4596#issuecomment-1137635703
@rdblue @lirui-apache thanks for confirming, so I think using 'rewriteManifest' and specifying the commit.manifest.target-size-byte to a reasonable size (so you have an even number of entries per manifest file), along with a system property: 'iceberg.worker.num-threads' to control how many manifests are read at once. I wonder if that will help that memory problem? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
