rbalamohan commented on PR #4512: URL: https://github.com/apache/iceberg/pull/4512#issuecomment-1091765832
Residuals are evaluated earlier to create the "expression" upfront. Computed expression (string) is much smaller than residual memory as it need not retain the literals. In the case I mentioned earlier, it retained 79000 LongLiterals which caused memory spikes as this was serialized in every split in Hive. With this change, expression is computed upfront and residuals need not be carried forward and thus reduces memory pressure. In the case you mentioned, expression gets evaluated to "true/false" upfront and thus reduces memory pressure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
