Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-68091244
I agree that it's a good idea to split this up.
For starters, I'm going to try splitting off only the fix for the
FileInputDStream test (SPARK-1600), since that's one of the flakiest tests, has
some somewhat-unique code changes (the file modification timestamp stuff) and
should be a relatively small PR to review by itself. I'll introduce the
StreamingTestWaiter class in that PR. Once we're done reviewing and merging
that, I'll move onto a PR to clean up all of the remaining uses of
`Thread.sleep()`. Some of those uses have not led to flaky tests, though, so I
think that splitting the change up and prioritizing based on the tests that are
known to be flaky will be a good way to reduce the review burden here.
Let's chat offline about the `saveAsHadoopFiles` fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
