mbutrovich commented on issue #5269: URL: https://github.com/apache/datafusion/issues/5269#issuecomment-4040758956
I think there's an opportunity for a DF52 postmortem when all is said and done, so I don't want to conflate the discussion too much. That said, from my perspective there were some stars that might have aligned to make DF52 a tough release from which we're still recovering: - Significant API churn with 52 (`SchemaAdapter`, etc.). This meant Comet couldn't test 52 before release, and maybe other downstream consumers have been slow to adopt it and find more issues in it. - Released during the holidays. - Slightly longer release window: DF 51.0 on November 19, 52.0 on January 12, but both were ~540 commits. All of that said, re: LTS releases: - We're struggling a bit to get 52.3 and 53 released at the same time, demonstrating the challenges of maintaining parallel releases. How would we handle cutting LTS releases while simultaneously cutting new major versions? - Selfishly: DataFusion Comet depends on projects (iceberg-rust) that may pin to an LTS. I am hoping to find a way to disentangle the iceberg-rust components that Comet needs from DataFusion dependency, but realistically they'd still be tied to the Arrow-rs version. If iceberg-rust pins to an LTS and Comet wants to keep up with DataFusion, then we'd likely have to fork iceberg-rust. - What would the backporting criteria be? Presumably it's only for regressions, but even that gets a little fuzzy w.r.t. performance differences. The hope would be that it wouldn't add significant reviewing workload, but we also want to be disciplined with what gets backported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
