Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21537 @gatorsmile I think this is just the partial adoption of what was proposed [here](https://github.com/apache/spark/pull/19813#issuecomment-354045400) and implemented in SPARK-24121 (https://github.com/apache/spark/pull/21193). Despite I agree with you that we should have created a design doc for this, I think now is a bit late for it. Indeed the base API is already there, since the design has mostly been done in SPARK-24121 (and not in the scope of this PR). What is missing now, though, is a throughout adoption. But there are already many PRs which based on that API: SPARK-22856, SPARK-23951. And the current codebase already partially adopts it. So I am not sure whether reverting this PR would be a solution: we should revert many if we want the change the current design. I think a design can be created anyway, at least in order to formalize what was discussed and agreed in the various PRs and have a single source of information. We can also do some modifications to the current design according to the comments which could come out discussing the design doc, but I see them more like further changes than reverting this, as it would mean reverting many things. The only last thing I'd suggest is that probably @viirya is the best one for creating the design doc, since the original proposal and implementation was done by him. Thanks.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org