Hadoop 3 isn't supported yet, not quite even in master. I think the
profile there exists for testing at the moment.
Others may know a way that it can work but don't think it would out of the box.
On Fri, Apr 5, 2019 at 12:53 PM akirillov wrote:
>
> Hi there! I'm trying to run Spark unit tests
You can always try. But Hadoop 3 is not yet supported by Spark.
On Fri, Apr 5, 2019 at 11:13 AM Anton Kirillov
wrote:
>
> Marcelo, Sean, thanks for the clarification. So in order to support Hadoop 3+
> the preferred way would be to use Hadoop-free builds and provide Hadoop
> dependencies in
Yes, you can try it, though I doubt that will 100% work. Have a look
at the "hadoop 3" JIRAs and PRs still in progress on master.
On Fri, Apr 5, 2019 at 1:14 PM Anton Kirillov
wrote:
>
> Marcelo, Sean, thanks for the clarification. So in order to support Hadoop 3+
> the preferred way would be
Marcelo, Sean, thanks for the clarification. So in order to support Hadoop
3+ the preferred way would be to use Hadoop-free builds and provide Hadoop
dependencies in the classpath, is that correct?
On Fri, Apr 5, 2019 at 10:57 AM Marcelo Vanzin wrote:
> The hadoop-3 profile doesn't really work
The hadoop-3 profile doesn't really work yet, not even on master.
That's being worked on still.
On Fri, Apr 5, 2019 at 10:53 AM akirillov wrote:
>
> Hi there! I'm trying to run Spark unit tests with the following profiles:
>
> And 'core' module fails with the following test failing with
>
Really sorry for the formatting. Here's the original message:
Hi there! I'm trying to run Spark unit tests with the following profiles:
./build/mvn test -Pmesos "-Phadoop-3.1" -Pnetlib-lgpl -Psparkr -Phive
-Phive-thriftserver
And 'core' module fails with the following test failing with
Hi there! I'm trying to run Spark unit tests with the following profiles:
And 'core' module fails with the following test failing with
NoClassDefFoundError:
In the meantime building a distribution works fine when running:
Also, there are no problems with running tests using Hadoop 2.7
