Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2882#issuecomment-60183139
Have you done a sbt/sbt clean? I did it.
Either ways, there is a real problem. The hdfs-minicluster depends on
hdfs-core stuff, a lot of whose dependencies have been excluded in the
core/pom.xml. That's because we dont want HDFS versions specific dependencies
to interfere with Spark build, This is okay since we dont need to run HDFS,
just interface with it. But in this case, since we need to run HDFS as a
minicluster, those exclusions are a problem.
There are probably two possible solutions
1. Based on maven profile, reinclude the excluded stuff in test and run the
mini-cluster. This is going to painfully complicated.
2. Not use mini-cluster, and rather have two different code paths for
local file system and hdfs. More complex code, but simpler to maintain, though
the unit test will not really test HDFS flushes correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
