Github user squito commented on the issue:
https://github.com/apache/spark/pull/20888
I think you're right about killing the wrong stage, but I don't think its
exactly what you've outlined. The original code doesn't try to kill a stage
with ID == 0 -- instead its just waiting until that volatile is set to
something > 0, and then proceeds. that seems to work fine, we do see that the
stage does get canceled OK once.
However, I think the problem is because the test [runs twice, with and
without
codegen](https://github.com/apache/spark/blob/4d37008c78d7d6b8f8a649b375ecc090700eca4f/sql/core/src/test/scala/org/apache/spark/sql/DataFrameRangeSuite.scala#L165).
The first time, it'll always wait to till the stage Id is set, because of
that `eventually { ... stageToKill > 0}`.
however, on the second iteration, that `stageToKill` may still be > 0 based
on the first iteration, not because its been set by the second iteration. So I
think you just need to reset the value to -1 between iterations.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
