jackye1995 commented on pull request #3685: URL: https://github.com/apache/iceberg/pull/3685#issuecomment-990623655
> So if we do revert, it probably only makes sense to move back `rewriteDataFiles` as I don't think any of the other actions had implementations Yes totally agree, I mentioned `CreateAction` and `SnapshotAction` because we also don't have alternative implementations for Spark 2.4 for those, but they don't have original implementations either. So that boils down to only the `rewriteDataFiles`. > For this we should have similar classes of Spark3BinPackStrategy, Spark3SortStrategy and then we need 2.4 version of DistributionAndOrderingUtils etc. For the future work, I really don't think we should spend effort in re-implementing the action in 2.4. We can mark that as deprecated and add a note "please move to Spark 3.x for the new action implementation". As Russell said it does not support sort strategy anyway, so it's only gonna be partial support even if implemented. I think only people who absolutely want that should try to backport, and hopefully this can push people to continuously move forward in Spark versions. That's the best way for us to support new Spark versions in the current architecture, by gradually dropping support of features in old versions, otherwise it would be too much burden for developers to maintain all the Spark versions, especially 2.4 because it's so different from the 3.x versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
