westonpace commented on issue #35333: URL: https://github.com/apache/arrow/issues/35333#issuecomment-1523656770
In general, we try and avoid planning and optimization in Acero. Building a full planner or optimizer is just too much work for the capacity we have. My hope is that someday a Substrait planner/optimizer will emerge but we are probably years out from that. Now, one could make the argument that this, similar to our implicit conversion in expressions, is a small enough addition and not going to cause us to fall off the slippery slope into planning. The implicit conversion that we do already handle, which @thisisnic mentions, is done when we are binding expressions to a schema. The rules there are actually encoded into the functions themselves (e.g. each function gets to decide which implicit casts are appropriate). As a result, it isn't something that is easily reusable in its current form. So I guess this is kind of a non-answer to the "can we do it?" question. I would say that it's doable, and if someone wanted to take this work on I would be willing to get it merged in. However, it is not a priority for me personally at the moment. Is there any way to do the explicit cast in dplyr? E.g. instead of making sure the schemas match in dplyr could you read in the data with the true schema and then cast id to id_64 (or id_32) right before the join? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
