tustvold commented on issue #2583: URL: https://github.com/apache/arrow-datafusion/issues/2583#issuecomment-1139477308
I've created https://github.com/apache/arrow-datafusion/pull/2632 to try to improve the workflow for arrow upgrades, as they're every two weeks. That said I feel something is a bit off with this, in my opinion Ballista should follow DataFusion upgrades, not lead them. Perhaps in lieu of releasing DataFusion more frequently, we could simply update the git pin within Ballista more frequently. This is what we do with IOx, and often prior to a more risky DataFusion PR we will create a candidate IOx PR to test it out. I think this strikes a pragmatic balance, leaving reviewer discretion as to when to upgrade what. When a breaking change is made to DataFusion, I think there should be an expectation that the contributor will help out with the corresponding changes to Ballista, but I think requiring the corresponding Ballista change in advance of changing DataFusion is a bit burdensome. My feeling is that at this stage in the arrow-rs and arrow-datafusion projects, which I'd describe as beta-ish, the priority should be iteration speed and trying out new ideas, and not on calcifying APIs for fear of breaking downstream code. I don't know, I would appreciate your thoughts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
