kuwii commented on PR #39190: URL: https://github.com/apache/spark/pull/39190#issuecomment-1399216960
Tried the example code in the [JIRA](https://issues.apache.org/jira/browse/SPARK-24415), and it is not affected by this change. Tasks showed in the stage are the same before and after this change.  Also, `numActiveStages` of that example is also `-1`. I think the reason we didn't notice it is because currently the property seems to be only available in `jobs` REST API, not web UI.  I've checked comments about these lines in that [PR](https://github.com/apache/spark/pull/22209). Code here is for handling stages metrics when `onStageCompleted` event is dropped somehow. But as mentioned in this PR, I think the logic to reduce `activeStages` here is incorrect, which should be removed when handling `onJobEnd` event.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
