Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21060
It looks to me this is a bug fix that can merit backporting, as
QueryExecutionListener is also marked as experimental,
In this case, I think @gatorsmile is worried one might have written a
listener that enumerates the possible function names, and that listener will
fail now with a new action name. I feel this is quite unlikely, but I also
appreciate @gatorsmile's concern for backward compatibility, and I've certainly
been wrong before when our fixes break existing workloads.
(On the spectrum of being extremely conservative to extremely liberal, I
think I'm in general more on the middle, whereas @gatorsmile probably leans
more to the conservative side. There isn't really anything wrong with this, and
it's good to have balancing forces in a project.)
How about this, @HyukjinKwon -- for the 2.3.x backport, add a config that
so it is possible to turn this off in production, if somebody actually has
their job failed because of this? It's a small delta from what this PR already
does, and that should alleviate the concerns @gatorsmile has. I'd also change
the function doc for onSuccess/onFailure to make it clear that we will add new
function names in the future, and users shouldn't expect a fixed list of
function names.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
