Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5554#issuecomment-94299581
Finally, figured out why `logError("Error in job generator", e)` won't fail
the test cases.
NotSerializableException also happens in the old codes, but it just forces
the Actor restart because it's not caught by any code. Then I use EventLoop to
replace Actor, so `NotSerializableException` will be caught and sent to the
`onError` method. If I use `logError("Error in job generator", e)`,
`NotSerializableException` is just logged and doesn't fail the tests. But if I
use `jobScheduler.reportError("Error in job generator", e)`, the error will be
propagated to the mail thread of the tests. That's why the tests fail.
`StreamingKMeans` must be `Serializable` since it will be used in the
closure.
So it's a bug that was hidden by Actor error mechanism. And now
`jobScheduler.reportError` exposes the bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
