alamb commented on issue #11513: URL: https://github.com/apache/datafusion/issues/11513#issuecomment-2239614500
Thank you @notfilippo -- I think this proposal is well thought out and makes a lot of sense to me. If we were to implement it I think the benefits for DataFusion would be enormous From my perspective, the use of Arrow types in logical planning in DataFusion (e.g. type coercion) has always been a little bit of an impedance mismatch. When there were just a few variants (e.g. `String`/`LargeString` and `Dictionary`) it was annoying but manageable. As Arrow evolves (e.g. to include `REEArray`, StringViewArray, etc) the mismatch is becoming more painful (e.g. https://github.com/apache/datafusion/issues/10220 is an example) > Care must be put in place not to introduce breaking changes for downstream crates and dependencies that build on top of DataFusion. I think breaking changes to the API is inevitable, but I think we can mange the pain through careful API thought and deprecation. More thoughts to follow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
