alamb commented on issue #9011: URL: https://github.com/apache/arrow-datafusion/issues/9011#issuecomment-1914569503
I am not sure what is going on here. Could it be that the `Pivot`/`Unpivot` operations somehow are configured to preserve all the columns (and thus the optimizer wouldn't be able to remove their use). This would be if the output of `Pivot::expressions()` refered to all the columns. Maybe we should extend their display to show what expressions are being used One thing that might help debug here, depending on how you are invoking the optimzer, is to look at how DataFusion uses this https://github.com/apache/arrow-datafusion/blob/1097dc0345079e81ba217ef23fd408a5c13c3d6c/datafusion/core/src/physical_optimizer/output_requirements.rs#L91 When it is optimizing internally, it uses `OutputRequirementExec` to describe what properties are needed at the output -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
