alamb commented on issue #5157: URL: https://github.com/apache/arrow-datafusion/issues/5157#issuecomment-1414339134
I looked at the trace and here are my observations: As @tustvold has said, if we can have `DFSchema` / `DFField` that don't copy the values https://github.com/apache/arrow-datafusion/issues/4680 around that would help immensly A large amount of the allocations come from `DFSchema::merge` -- see https://github.com/apache/arrow-datafusion/blob/224c682101949da57aebc36e92e5a881ef3040d4/datafusion/common/src/dfschema.rs#L135-L151  And a large part of that is how it ignores errors with `.ok()` where were quite expensive to produce It also appears there is copying going on in unwrap_cast_in_comparison and common subexpr eliminiate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
