zeodtr commented on issue #7698: URL: https://github.com/apache/arrow-datafusion/issues/7698#issuecomment-1807496338
`DFField::qualified_name()` is another performance bottleneck. For a simple SELECT many-columns query for a table that has 3617 columns, the function took 73% of the total planning time. The problem is that the function calls `format!` (which is expensive) every time it is called when it has a qualifier. When I changed the DFField to have a member variable that has precomputed `qualified_name` and to return its clone() when `qualified_name()` is called, the planning became 2~3 times faster. But still, `clone()`ing the precomputed qualified_name takes a significant percentage (~30%? I didn't measure it with the same condition) of the total planning time. (I've tested it with DataFusion 31.0.0) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
