jinchengchenghh opened a new issue, #7359: URL: https://github.com/apache/incubator-gluten/issues/7359
### Description The UDF may cause the total project input col to row converter and the output row to column. For example, hash(udf(col0)), col1, col2, col3, col4, with partial project, we can only convert col0 as column to row or column to arrow as input, and convert udf(col0) as `Alias` partitalProject1_, then row to column. ProjectExecTransformer will be hash(partitalProject1_), col1, col2, col3, col4,partitalProject1_. Partial project will supply two path, velox columnar to row as input and row to velox columnar as output by UnsafeProjection. velox columnar to arrow as input and arrow to velox columnar as output by MutableProjection. The test results indicates it can help accelerate some queries 30%. The performance improvement varies with queries. Possible optimization: 1. Mutable Projection will write to buffer firstly, we can implement as mutable projection but directedly write to arrow, and can support variable length type such as string; 2. Arrow to Velox conversion is also expense, after Velox Java API, we may get more benefit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
