Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21060
> withCallback was added in Spark 1.6 release
https://issues.apache.org/jira/browse/SPARK-11068 Since then, my understanding
is we never clearly define which should be part of withCallback. Thus, it is
hard to say this is a bug fix.
The callback works for `collect` in R and Scala but Python doesn't. I think
we should at least match the behaviour. I wonder why it's hard to say a bug
when `collect` is detected in some APIs but not in some APIs.
> We hit the similar issue in #18064. At that time, we did not backport the
PR to the previous releases too.
That's because the change was big and invasive. I wouldn't backport it too;
however, this fix is relatively small.
> Thus, I do not think we should make an exception for this PR just because
the customers of @HyukjinKwon hit this issue
It's not because my customers but I am saying it fixes an actual usecase
and it affects actual users.
> If we make an exception, it becomes harder to decide which PRs are
qualified for a backport.
I think we usually use committer's judgement when we make an exception. I
already have been seeing many backports that actually causes behaviour changes
and I did this because it looks being backported in general. This is the reason
why we should formally document it if this is actually the rule.
What I am less sure is, why this one specifically prompted you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
