Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5473#issuecomment-95313611
You have to add tests to the StreamingJobProgressListener to test the use
thread-local properties to combine streaming and spark event information. This
is very subtle code and can silently break if someone change the behavior of
thread-local properties in spark job properties. Also test the expected number
of output ids, and Spark job ids using a program with multiple output ops and
multiple Spark actions/jobs in each output op.
You can test these in the StreamingJobProgressListenerSuite in the same
way you test the batch and receiver events - by adding spark events in the mix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
