Re: java.lang.ClassNotFoundException - spark on mesos

[bo...@shopify.com]

I have opened a PR for discussion on the apache/spark repository
https://github.com/apache/spark/pull/620

There is certainly a classLoader problem in the way Mesos and Spark operate,
I'm not sure what caused it to suddenly stop working so I'd like to open the
discussion there



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-tp3510p5245.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: java.lang.ClassNotFoundException - spark on mesos

[Bharath Bhushan]

I tried several things in order to get 1.0.0 git tree to work with 
mesos. All my efforts failed. I could run spark 0.9.0 on mesos but not 
spark 1.0.0. Please suggest any other things I can try.


1. Change project/SparkBuild.scala to use mesos 0.17.0 and then 
make_distribution.sh.


2. Try building with maven. Threw the following error:
[INFO] Spark Project Parent POM .. SUCCESS 
[7:55.796s]

[INFO] Spark Project Core  FAILURE [7.209s]
...
[ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one 
of its
dependencies could not be resolved: Failed to read artifact descriptor 
for org.a
pache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find 
artifact org.a

pache:apache:pom:13 -> [Help 1]

3. Change pom.xml to use mesos 0.17.0 and protobuf 2.5.0 and then 
make_distribution.sh


4. Change pom.xml to use mesos 0.17.0 and leave protobuf at 2.4.1 and 
then make_distribution.sh


5. Change pom.xml to use mesos 0.17.0 and remove the protobuf line and 
then make_distribution.sh


Question: Is the pom.xml modification picked up when building with sbt 
and maven or only when building with maven?


Thanks

On 01/04/14 11:04 am, Bharath Bhushan wrote:

Another problem I noticed is that the current 1.0.0 git tree still gives me the 
ClassNotFoundException. I see that the SPARK-1052 is already fixed there. I 
then modified the pom.xml for mesos and protobuf and that still gave the 
ClassNotFoundException. I also tried modifying pom.xml only for mesos and that 
fails too. So I have no way of running the 1.0.0 git tree spark on mesos yet.

Thanks.

On 01-Apr-2014, at 3:28 am, deric  wrote:


Which repository do you use?

The issue should be fixed in 0.9.1 and 1.0.0

https://spark-project.atlassian.net/browse/SPARK-1052


There's an old repository

https://github.com/apache/incubator-spark

and as Spark become one of top level projects, it was moved to new repo:

https://github.com/apache/spark

The 0.9.1 version hasn't been released yet, so you should get it from the
new git repo.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-tp3510p3551.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: java.lang.ClassNotFoundException - spark on mesos

[Bharath Bhushan]

Another problem I noticed is that the current 1.0.0 git tree still gives me the 
ClassNotFoundException. I see that the SPARK-1052 is already fixed there. I 
then modified the pom.xml for mesos and protobuf and that still gave the 
ClassNotFoundException. I also tried modifying pom.xml only for mesos and that 
fails too. So I have no way of running the 1.0.0 git tree spark on mesos yet.

Thanks.

On 01-Apr-2014, at 3:28 am, deric  wrote:

> Which repository do you use?
> 
> The issue should be fixed in 0.9.1 and 1.0.0
> 
> https://spark-project.atlassian.net/browse/SPARK-1052
>   
> 
> There's an old repository 
> 
> https://github.com/apache/incubator-spark
> 
> and as Spark become one of top level projects, it was moved to new repo:
> 
> https://github.com/apache/spark
> 
> The 0.9.1 version hasn't been released yet, so you should get it from the
> new git repo.
> 
> 
> 
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-tp3510p3551.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: java.lang.ClassNotFoundException - spark on mesos

[Bharath Bhushan]

I was talking about the protobuf version issue as not fixed. I could not find 
any reference to the problem or the fix.

Reg. SPARK-1052, I could pull in the fix into my 0.9.0 tree (from the tar ball 
on the website) and I see the fix in the latest git.

Thanks

On 01-Apr-2014, at 3:28 am, deric  wrote:

> Which repository do you use?
> 
> The issue should be fixed in 0.9.1 and 1.0.0
> 
> https://spark-project.atlassian.net/browse/SPARK-1052
>   
> 
> There's an old repository 
> 
> https://github.com/apache/incubator-spark
> 
> and as Spark become one of top level projects, it was moved to new repo:
> 
> https://github.com/apache/spark
> 
> The 0.9.1 version hasn't been released yet, so you should get it from the
> new git repo.
> 
> 
> 
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-tp3510p3551.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: java.lang.ClassNotFoundException - spark on mesos

[deric]

Which repository do you use?

The issue should be fixed in 0.9.1 and 1.0.0

https://spark-project.atlassian.net/browse/SPARK-1052
  

There's an old repository 

https://github.com/apache/incubator-spark

and as Spark become one of top level projects, it was moved to new repo:

https://github.com/apache/spark

The 0.9.1 version hasn't been released yet, so you should get it from the
new git repo.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-tp3510p3551.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: java.lang.ClassNotFoundException - spark on mesos

[Bharath Bhushan]

Your suggestion took me past the ClassNotFoundException. I then hit 
akka.actor.ActorNotFound exception. I patched in PR 568 into my 0.9.0 spark 
codebase and everything worked.

So thanks a lot, Tim. Is there a JIRA/PR for the protobuf issue? Why is it not 
fixed in the latest git tree?

Thanks.

On 31-Mar-2014, at 11:30 pm, Tim St Clair  wrote:

> It sounds like the protobuf issue. 
> 
> So FWIW, You might want to try updating the 0.9.0 w/pom mods for mesos & 
> protobuf. 
> 
> mesos 0.17.0 & protobuf 2.5   
> 
> Cheers,
> Tim
> 
> - Original Message -
>> From: "Bharath Bhushan" 
>> To: user@spark.apache.org
>> Sent: Monday, March 31, 2014 9:46:32 AM
>> Subject: Re: java.lang.ClassNotFoundException - spark on mesos
>> 
>> I tried 0.9.0 and the latest git tree of spark. For mesos, I tried 0.17.0 and
>> the latest git tree.
>> 
>> Thanks
>> 
>> 
>> On 31-Mar-2014, at 7:24 pm, Tim St Clair  wrote:
>> 
>>> What versions are you running?
>>> 
>>> There is a known protobuf 2.5 mismatch, depending on your versions.
>>> 
>>> Cheers,
>>> Tim
>>> 
>>> - Original Message -
>>>> From: "Bharath Bhushan" 
>>>> To: user@spark.apache.org
>>>> Sent: Monday, March 31, 2014 8:16:19 AM
>>>> Subject: java.lang.ClassNotFoundException - spark on mesos
>>>> 
>>>> I am facing different kinds of java.lang.ClassNotFoundException when
>>>> trying
>>>> to run spark on mesos. One error has to do with
>>>> org.apache.spark.executor.MesosExecutorBackend. Another has to do with
>>>> org.apache.spark.serializer.JavaSerializer. I see other people complaining
>>>> about similar issues.
>>>> 
>>>> I tried with different version of spark distribution - 0.9.0 and
>>>> 1.0.0-SNAPSHOT and faced the same problem. I think the reason for this is
>>>> is
>>>> related to the error below.
>>>> 
>>>> $ jar -xf spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
>>>> java.io.IOException: META-INF/license : could not create directory
>>>>   at sun.tools.jar.Main.extractFile(Main.java:907)
>>>>   at sun.tools.jar.Main.extract(Main.java:850)
>>>>   at sun.tools.jar.Main.run(Main.java:240)
>>>>   at sun.tools.jar.Main.main(Main.java:1147)
>>>> 
>>>> This error happens with all the jars that I created. But the classes that
>>>> are
>>>> already generated is different in the different cases. If JavaSerializer
>>>> is
>>>> not already extracted before encountering META-INF/license, then that
>>>> class
>>>> is not found during execution. If MesosExecutorBackend is not found, then
>>>> that class shows up in the mesos slave error logs. Can someone confirm if
>>>> this is a valid cause for the problem I am seeing? Any way I can debug
>>>> this
>>>> further?
>>>> 
>>>> — Bharath
>>> 
>>> --
>>> Cheers,
>>> Tim
>>> Freedom, Features, Friends, First -> Fedora
>>> https://fedoraproject.org/wiki/SIGs/bigdata
>> 
>> 
> 
> -- 
> Cheers,
> Tim
> Freedom, Features, Friends, First -> Fedora
> https://fedoraproject.org/wiki/SIGs/bigdata

Re: java.lang.ClassNotFoundException - spark on mesos

[Tim St Clair]

It sounds like the protobuf issue. 

So FWIW, You might want to try updating the 0.9.0 w/pom mods for mesos & 
protobuf. 

mesos 0.17.0 & protobuf 2.5   

Cheers,
Tim

- Original Message -
> From: "Bharath Bhushan" 
> To: user@spark.apache.org
> Sent: Monday, March 31, 2014 9:46:32 AM
> Subject: Re: java.lang.ClassNotFoundException - spark on mesos
> 
> I tried 0.9.0 and the latest git tree of spark. For mesos, I tried 0.17.0 and
> the latest git tree.
> 
> Thanks
> 
> 
> On 31-Mar-2014, at 7:24 pm, Tim St Clair  wrote:
> 
> > What versions are you running?
> > 
> > There is a known protobuf 2.5 mismatch, depending on your versions.
> > 
> > Cheers,
> > Tim
> > 
> > - Original Message -
> >> From: "Bharath Bhushan" 
> >> To: user@spark.apache.org
> >> Sent: Monday, March 31, 2014 8:16:19 AM
> >> Subject: java.lang.ClassNotFoundException - spark on mesos
> >> 
> >> I am facing different kinds of java.lang.ClassNotFoundException when
> >> trying
> >> to run spark on mesos. One error has to do with
> >> org.apache.spark.executor.MesosExecutorBackend. Another has to do with
> >> org.apache.spark.serializer.JavaSerializer. I see other people complaining
> >> about similar issues.
> >> 
> >> I tried with different version of spark distribution - 0.9.0 and
> >> 1.0.0-SNAPSHOT and faced the same problem. I think the reason for this is
> >> is
> >> related to the error below.
> >> 
> >> $ jar -xf spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
> >> java.io.IOException: META-INF/license : could not create directory
> >>at sun.tools.jar.Main.extractFile(Main.java:907)
> >>at sun.tools.jar.Main.extract(Main.java:850)
> >>at sun.tools.jar.Main.run(Main.java:240)
> >>at sun.tools.jar.Main.main(Main.java:1147)
> >> 
> >> This error happens with all the jars that I created. But the classes that
> >> are
> >> already generated is different in the different cases. If JavaSerializer
> >> is
> >> not already extracted before encountering META-INF/license, then that
> >> class
> >> is not found during execution. If MesosExecutorBackend is not found, then
> >> that class shows up in the mesos slave error logs. Can someone confirm if
> >> this is a valid cause for the problem I am seeing? Any way I can debug
> >> this
> >> further?
> >> 
> >> — Bharath
> > 
> > --
> > Cheers,
> > Tim
> > Freedom, Features, Friends, First -> Fedora
> > https://fedoraproject.org/wiki/SIGs/bigdata
> 
> 

-- 
Cheers,
Tim
Freedom, Features, Friends, First -> Fedora
https://fedoraproject.org/wiki/SIGs/bigdata

Re: java.lang.ClassNotFoundException - spark on mesos

[Bharath Bhushan]

I tried 0.9.0 and the latest git tree of spark. For mesos, I tried 0.17.0 and 
the latest git tree.

Thanks


On 31-Mar-2014, at 7:24 pm, Tim St Clair  wrote:

> What versions are you running?  
> 
> There is a known protobuf 2.5 mismatch, depending on your versions. 
> 
> Cheers,
> Tim
> 
> - Original Message -
>> From: "Bharath Bhushan" 
>> To: user@spark.apache.org
>> Sent: Monday, March 31, 2014 8:16:19 AM
>> Subject: java.lang.ClassNotFoundException - spark on mesos
>> 
>> I am facing different kinds of java.lang.ClassNotFoundException when trying
>> to run spark on mesos. One error has to do with
>> org.apache.spark.executor.MesosExecutorBackend. Another has to do with
>> org.apache.spark.serializer.JavaSerializer. I see other people complaining
>> about similar issues.
>> 
>> I tried with different version of spark distribution - 0.9.0 and
>> 1.0.0-SNAPSHOT and faced the same problem. I think the reason for this is is
>> related to the error below.
>> 
>> $ jar -xf spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
>> java.io.IOException: META-INF/license : could not create directory
>>at sun.tools.jar.Main.extractFile(Main.java:907)
>>at sun.tools.jar.Main.extract(Main.java:850)
>>at sun.tools.jar.Main.run(Main.java:240)
>>at sun.tools.jar.Main.main(Main.java:1147)
>> 
>> This error happens with all the jars that I created. But the classes that are
>> already generated is different in the different cases. If JavaSerializer is
>> not already extracted before encountering META-INF/license, then that class
>> is not found during execution. If MesosExecutorBackend is not found, then
>> that class shows up in the mesos slave error logs. Can someone confirm if
>> this is a valid cause for the problem I am seeing? Any way I can debug this
>> further?
>> 
>> — Bharath
> 
> -- 
> Cheers,
> Tim
> Freedom, Features, Friends, First -> Fedora
> https://fedoraproject.org/wiki/SIGs/bigdata

Re: java.lang.ClassNotFoundException - spark on mesos

[Tim St Clair]

What versions are you running?  

There is a known protobuf 2.5 mismatch, depending on your versions. 

Cheers,
Tim

- Original Message -
> From: "Bharath Bhushan" 
> To: user@spark.apache.org
> Sent: Monday, March 31, 2014 8:16:19 AM
> Subject: java.lang.ClassNotFoundException - spark on mesos
> 
> I am facing different kinds of java.lang.ClassNotFoundException when trying
> to run spark on mesos. One error has to do with
> org.apache.spark.executor.MesosExecutorBackend. Another has to do with
> org.apache.spark.serializer.JavaSerializer. I see other people complaining
> about similar issues.
> 
> I tried with different version of spark distribution - 0.9.0 and
> 1.0.0-SNAPSHOT and faced the same problem. I think the reason for this is is
> related to the error below.
> 
> $ jar -xf spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
> java.io.IOException: META-INF/license : could not create directory
> at sun.tools.jar.Main.extractFile(Main.java:907)
> at sun.tools.jar.Main.extract(Main.java:850)
> at sun.tools.jar.Main.run(Main.java:240)
> at sun.tools.jar.Main.main(Main.java:1147)
> 
> This error happens with all the jars that I created. But the classes that are
> already generated is different in the different cases. If JavaSerializer is
> not already extracted before encountering META-INF/license, then that class
> is not found during execution. If MesosExecutorBackend is not found, then
> that class shows up in the mesos slave error logs. Can someone confirm if
> this is a valid cause for the problem I am seeing? Any way I can debug this
> further?
> 
> — Bharath

-- 
Cheers,
Tim
Freedom, Features, Friends, First -> Fedora
https://fedoraproject.org/wiki/SIGs/bigdata

java.lang.ClassNotFoundException - spark on mesos

[Bharath Bhushan]

I am facing different kinds of java.lang.ClassNotFoundException when trying to 
run spark on mesos. One error has to do with 
org.apache.spark.executor.MesosExecutorBackend. Another has to do with 
org.apache.spark.serializer.JavaSerializer. I see other people complaining 
about similar issues.

I tried with different version of spark distribution - 0.9.0 and 1.0.0-SNAPSHOT 
and faced the same problem. I think the reason for this is is related to the 
error below.

$ jar -xf spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
java.io.IOException: META-INF/license : could not create directory
at sun.tools.jar.Main.extractFile(Main.java:907)
at sun.tools.jar.Main.extract(Main.java:850)
at sun.tools.jar.Main.run(Main.java:240)
at sun.tools.jar.Main.main(Main.java:1147)

This error happens with all the jars that I created. But the classes that are 
already generated is different in the different cases. If JavaSerializer is not 
already extracted before encountering META-INF/license, then that class is not 
found during execution. If MesosExecutorBackend is not found, then that class 
shows up in the mesos slave error logs. Can someone confirm if this is a valid 
cause for the problem I am seeing? Any way I can debug this further?

— Bharath