Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3695#issuecomment-67036493
> [...] conflicts reported in SPARK-4826 was likely caused by multiple
tests running at the same time and using the same filename.
Here's what I find confusing, though: the Jenkins master builds (and
branch-1.2 builds) don't have any parallelism within a single build: only one
thread of control should be executing in a given instance of
`WriteAheadLogBackedBlockRDDSuite`. There might be multiple instances of
`WriteAheadLogBackedBlockRDDSuite` executing in different Jenkins builds, but I
would expect that the different Jenkins builds would have different values of
`Files.createTempDir()` and hence would write to different output locations.
Are you saying that within the same Jenkins build / instance of
`WriteAheadLogBackedBlockRDDSuite`, we are having two separate tests that are
choosing to write to the same subdirectory / file inside of
`Files.createTempDir()`?
Maybe something is implicitly calling `Random.setSeed()` between test
cases, causing the tests to pick the same subdirectory. If that's the case, we
might be able to fix things by moving the directory deletion call to a
`afterEach()` method or by using `Files.createTempFile` within that temp
directory in order to pick the filename.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
