szlta commented on PR #4512: URL: https://github.com/apache/iceberg/pull/4512#issuecomment-1096956287
Almost.. we would still be running residuals, they'd just take a different way to their final evaluation. So right we're transferring the filter expression within every split. This expression hangs on every baseFileScanTask.residuals.expr in its original form i.e. un-evaluated against partition info. (This actually makes it less of a "residual" so to speak.) If the expression is big and we have thousands of splits, this leads to OOM at the point where every split is getting serialized. (A limitation of TezAM is that streamed serialization is not possible..) On the executors, after deserialization we have these baseFileScanTasks along with the original filter expressions which get evaluated against the partition value of the task when we call task.residual(), and thus it becomes an actual "residual" expression to be pushed down to file format libs. So far this is how it works currently. My proposal is that since the expressions that get serialized into the splits are always all the same for a given job, Hive should call ignoreResiduals() to make Iceberg set Expressions.alwaysTrue() in the baseFileScanTask instances. Therefore we don't serialize the original filter expressions into every split. We can do this, because the original filter can be retrieved from the Hive job conf anyway. Then with a little trick on the executor processes, we won't take task.residual(), but instead look for the original filter in the job conf, evaluate it according to the current task (spec and partition), and pass the resulting real residual filter to file formats/record readers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
