attilapiros commented on pull request #34672: URL: https://github.com/apache/spark/pull/34672#issuecomment-1011123883
Actually I need help from all of you. I checked #30763 and the current code. And as I see now it is even harder to create an API interface because of some extra features integrated into master (like the pushed based shuffle). So I thought about the problem (how we can get rid of the extra methods polluting generic interfaces) and came up with something. When the feature is controlled by a feature flag we are safe in runtime (these methods are not used). My intention is to do minimal changes (in risk and size) regarding the Spark internal features (plugins in this sense are secondary). So my solution is to cast into specific implementations where this extra functionality needed. I can show it via an example how I cleaned up `ShuffleBlockResolver`: https://github.com/apache/spark/pull/35180 If this fine we should continue with `MapOutputTracker` which I think should be part of Shuffle API with a meaningful base implementation. My plan to progress with baby steps to make the reviewers life easier. @tgravescs, @mridulm, @yangwwei. @hiboyang: WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
