Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21060
I am a bit puzzled because `QueryExecutionListener` should call the
callback for actions and `collect` triggers it in Scala and R but it doesn't in
PySpark specifically. It sounds a bug and this fix is relatively
straightforward. The previous behaviour was it was not being called which
didn't make sense.
I agree that it's discouraged to make a behaviour change to the maintenance
release, sure. However, I was thinking it makes sense to backport if the fix is
not complicated and looks a bug quite clearly. I think we shouldn't say it's
improvement in this case.
Were actual apps or test cases broken somewhere?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
