Re: Running spark from Eclipse and then Jar

[Iman Mohtashemi]

Oh thanks! I'll take a look

On Sat, Dec 10, 2016 at 11:37 AM Md. Rezaul Karim <
rezaul.ka...@insight-centre.org> wrote:

> Hello Iman,
>
> Finally, I managed to solve the problem. I had been experiencing the
> problem because of the locking issue in the "*metastore_db*" under the
> project tree on Eclipse.
>
> If you see the project tree, under the "*metastore_db*" folder you should
> see a file named "*db.lck*" file which was preventing the jar to be
> executed from the command line.
>
> I just deleted that file, packaged my project as jar again and finally the
> problem resolved.
>
>
>
>
> Regards,
> _
> *Md. Rezaul Karim* BSc, MSc
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> 
>
> On 8 December 2016 at 01:15, Iman Mohtashemi 
> wrote:
>
> yes exactly. I run mine fine in Eclipse but when I run it from a
> corresponding jar I get the same error!
>
> On Wed, Dec 7, 2016 at 5:04 PM Md. Rezaul Karim <
> rezaul.ka...@insight-centre.org> wrote:
>
> I believe, it's not about the location (i.e., local machine or HDFS) but
> it's all about the format of the input file. For example, I am getting the
> following error while trying to read an input file in libsvm format:
>
> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to
> find data  source: libsvm. *
>
> The application works fine on Eclipse. However, while packaging the
> corresponding jar file, I am getting the above error which is really weird!
>
>
>
> Regards,
> _
> *Md. Rezaul Karim* BSc, MSc
>
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> 
>
> On 7 December 2016 at 23:39, Iman Mohtashemi 
> wrote:
>
> No but I tried that too and still didn't work. Where are the files being
> read from? From the local machine or HDFS? Do I need to get the files to
> HDFS first? In Eclipse I just point to the location of the directory?
>
> On Wed, Dec 7, 2016 at 3:34 PM Md. Rezaul Karim <
> rezaul.ka...@insight-centre.org> wrote:
>
> Hi,
>
> You should prepare your jar file (from your Spark application written in
> Java) with all the necessary dependencies. You can create a Maven project
> on Eclipse by specifying the dependencies in a Maven friendly pom.xml file.
>
> For building the jar with the dependencies and *main class (since you are
> getting the **ClassNotFoundException)* your pom.xml should contain the
> following in the *build *tag (example main class is marked in Red color):
>
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-eclipse-plugin
> 2.9
> 
> true
> false
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-compiler-plugin
> 3.5.1
> 
> ${jdk.version}
> ${jdk.version}
> 
> 
> 
> org.apache.maven.plugins
> maven-shade-plugin
> 2.4.3
> 
> true
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-assembly-plugin
> 2.4.1
> 
> 
> 
>
> jar-with-dependencies
> 
> 
> 
> 
>
> com.example.RandomForest.SongPrediction
> 
> 
>
> 
>
> oozie.launcher.mapreduce.job.user.classpath.first
> true
> 
>
> 
> 
> 
> make-assembly
> 
> package
> 
> single
> 
> 
> 
> 
> 
> 
>
>
> An example pom.xml file has been attached for your reference. Feel free to
> reuse it.
>
>
> Regards,
> _
> *Md. Rezaul Karim,* BSc, MSc
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> 
>
> On 7 December 2016 at 23:18, im281  wrote:
>
> Hello,
> I have a simple word count example in Java and I can run this 

Re: Running spark from Eclipse and then Jar

[Md. Rezaul Karim]

Hello Iman,

Finally, I managed to solve the problem. I had been experiencing the
problem because of the locking issue in the "*metastore_db*" under the
project tree on Eclipse.

If you see the project tree, under the "*metastore_db*" folder you should
see a file named "*db.lck*" file which was preventing the jar to be
executed from the command line.

I just deleted that file, packaged my project as jar again and finally the
problem resolved.




Regards,
_
*Md. Rezaul Karim* BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web: http://www.reza-analytics.eu/index.html


On 8 December 2016 at 01:15, Iman Mohtashemi 
wrote:

> yes exactly. I run mine fine in Eclipse but when I run it from a
> corresponding jar I get the same error!
>
> On Wed, Dec 7, 2016 at 5:04 PM Md. Rezaul Karim <
> rezaul.ka...@insight-centre.org> wrote:
>
>> I believe, it's not about the location (i.e., local machine or HDFS) but
>> it's all about the format of the input file. For example, I am getting the
>> following error while trying to read an input file in libsvm format:
>>
>> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to
>> find data  source: libsvm. *
>>
>> The application works fine on Eclipse. However, while packaging the
>> corresponding jar file, I am getting the above error which is really weird!
>>
>>
>>
>> Regards,
>> _
>> *Md. Rezaul Karim* BSc, MSc
>>
>> PhD Researcher, INSIGHT Centre for Data Analytics
>> National University of Ireland, Galway
>> IDA Business Park, Dangan, Galway, Ireland
>> Web: http://www.reza-analytics.eu/index.html
>> 
>>
>> On 7 December 2016 at 23:39, Iman Mohtashemi 
>> wrote:
>>
>> No but I tried that too and still didn't work. Where are the files being
>> read from? From the local machine or HDFS? Do I need to get the files to
>> HDFS first? In Eclipse I just point to the location of the directory?
>>
>> On Wed, Dec 7, 2016 at 3:34 PM Md. Rezaul Karim <
>> rezaul.ka...@insight-centre.org> wrote:
>>
>> Hi,
>>
>> You should prepare your jar file (from your Spark application written in
>> Java) with all the necessary dependencies. You can create a Maven project
>> on Eclipse by specifying the dependencies in a Maven friendly pom.xml file.
>>
>> For building the jar with the dependencies and *main class (since you
>> are getting the **ClassNotFoundException)* your pom.xml should contain
>> the following in the *build *tag (example main class is marked in Red
>> color):
>>
>> 
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-eclipse-plugin
>> 2.9
>> 
>> true
>> false
>> 
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-compiler-plugin
>> 3.5.1
>> 
>> ${jdk.version}
>> ${jdk.version}
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-shade-plugin
>> 2.4.3
>> 
>> true
>> 
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-assembly-plugin
>> 2.4.1
>> 
>> 
>> 
>> jar-with-
>> dependencies
>> 
>> 
>> 
>> 
>> com.example.
>> RandomForest.SongPrediction
>> 
>> 
>>
>> 
>> oozie.launcher.
>> mapreduce.job.user.classpath.first
>> true
>> 
>>
>> 
>> 
>> 
>> make-assembly
>> 
>> package
>> 
>> single
>> 
>> 
>> 
>> 
>> 
>> 
>>
>>
>> An example pom.xml file has been attached for your reference. Feel free
>> to reuse it.
>>
>>
>> Regards,
>> _
>> *Md. Rezaul Karim,* BSc, MSc
>> PhD Researcher, INSIGHT Centre for Data Analytics
>> National University of Ireland, Galway
>> IDA Business Park, Dangan, Galway, Ireland
>> Web: http://www.reza-analytics.eu/index.html
>> 
>>
>> On 7 December 2016 at 23:18, im281  wrote:
>>
>> Hello,
>> I have a simple 

Re: Running spark from Eclipse and then Jar

[Iman Mohtashemi]

yes exactly. I run mine fine in Eclipse but when I run it from a
corresponding jar I get the same error!

On Wed, Dec 7, 2016 at 5:04 PM Md. Rezaul Karim <
rezaul.ka...@insight-centre.org> wrote:

> I believe, it's not about the location (i.e., local machine or HDFS) but
> it's all about the format of the input file. For example, I am getting the
> following error while trying to read an input file in libsvm format:
>
> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to
> find data  source: libsvm. *
>
> The application works fine on Eclipse. However, while packaging the
> corresponding jar file, I am getting the above error which is really weird!
>
>
>
> Regards,
> _
> *Md. Rezaul Karim* BSc, MSc
>
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> 
>
> On 7 December 2016 at 23:39, Iman Mohtashemi 
> wrote:
>
> No but I tried that too and still didn't work. Where are the files being
> read from? From the local machine or HDFS? Do I need to get the files to
> HDFS first? In Eclipse I just point to the location of the directory?
>
> On Wed, Dec 7, 2016 at 3:34 PM Md. Rezaul Karim <
> rezaul.ka...@insight-centre.org> wrote:
>
> Hi,
>
> You should prepare your jar file (from your Spark application written in
> Java) with all the necessary dependencies. You can create a Maven project
> on Eclipse by specifying the dependencies in a Maven friendly pom.xml file.
>
> For building the jar with the dependencies and *main class (since you are
> getting the **ClassNotFoundException)* your pom.xml should contain the
> following in the *build *tag (example main class is marked in Red color):
>
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-eclipse-plugin
> 2.9
> 
> true
> false
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-compiler-plugin
> 3.5.1
> 
> ${jdk.version}
> ${jdk.version}
> 
> 
> 
> org.apache.maven.plugins
> maven-shade-plugin
> 2.4.3
> 
> true
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-assembly-plugin
> 2.4.1
> 
> 
> 
>
> jar-with-dependencies
> 
> 
> 
> 
>
> com.example.RandomForest.SongPrediction
> 
> 
>
> 
>
> oozie.launcher.mapreduce.job.user.classpath.first
> true
> 
>
> 
> 
> 
> make-assembly
> 
> package
> 
> single
> 
> 
> 
> 
> 
> 
>
>
> An example pom.xml file has been attached for your reference. Feel free to
> reuse it.
>
>
> Regards,
> _
> *Md. Rezaul Karim,* BSc, MSc
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> 
>
> On 7 December 2016 at 23:18, im281  wrote:
>
> Hello,
> I have a simple word count example in Java and I can run this in Eclipse
> (code at the bottom)
>
> I then create a jar file from it and try to run it from the cmd
>
>
> java -jar C:\Users\Owner\Desktop\wordcount.jar Data/testfile.txt
>
> But I get this error?
>
> I think the main error is:
> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to
> find
> data source: text*
>
> Any advise on how to run this jar file in spark would be appreciated
>
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/12/07 15:16:41 INFO SparkContext: Running Spark version 2.0.2
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users  with view permissions: 

Re: Running spark from Eclipse and then Jar

[Md. Rezaul Karim]

I believe, it's not about the location (i.e., local machine or HDFS) but
it's all about the format of the input file. For example, I am getting the
following error while trying to read an input file in libsvm format:

*Exception in thread "main" java.lang.ClassNotFoundException: Failed to
find data  source: libsvm. *

The application works fine on Eclipse. However, while packaging the
corresponding jar file, I am getting the above error which is really weird!



Regards,
_
*Md. Rezaul Karim* BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web: http://www.reza-analytics.eu/index.html


On 7 December 2016 at 23:39, Iman Mohtashemi 
wrote:

> No but I tried that too and still didn't work. Where are the files being
> read from? From the local machine or HDFS? Do I need to get the files to
> HDFS first? In Eclipse I just point to the location of the directory?
>
> On Wed, Dec 7, 2016 at 3:34 PM Md. Rezaul Karim <
> rezaul.ka...@insight-centre.org> wrote:
>
>> Hi,
>>
>> You should prepare your jar file (from your Spark application written in
>> Java) with all the necessary dependencies. You can create a Maven project
>> on Eclipse by specifying the dependencies in a Maven friendly pom.xml file.
>>
>> For building the jar with the dependencies and *main class (since you
>> are getting the **ClassNotFoundException)* your pom.xml should contain
>> the following in the *build *tag (example main class is marked in Red
>> color):
>>
>> 
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-eclipse-plugin
>> 2.9
>> 
>> true
>> false
>> 
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-compiler-plugin
>> 3.5.1
>> 
>> ${jdk.version}
>> ${jdk.version}
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-shade-plugin
>> 2.4.3
>> 
>> true
>> 
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-assembly-plugin
>> 2.4.1
>> 
>> 
>> 
>> jar-with-
>> dependencies
>> 
>> 
>> 
>> 
>> com.example.
>> RandomForest.SongPrediction
>> 
>> 
>>
>> 
>> oozie.launcher.
>> mapreduce.job.user.classpath.first
>> true
>> 
>>
>> 
>> 
>> 
>> make-assembly
>> 
>> package
>> 
>> single
>> 
>> 
>> 
>> 
>> 
>> 
>>
>>
>> An example pom.xml file has been attached for your reference. Feel free
>> to reuse it.
>>
>>
>> Regards,
>> _
>> *Md. Rezaul Karim,* BSc, MSc
>> PhD Researcher, INSIGHT Centre for Data Analytics
>> National University of Ireland, Galway
>> IDA Business Park, Dangan, Galway, Ireland
>> Web: http://www.reza-analytics.eu/index.html
>> 
>>
>> On 7 December 2016 at 23:18, im281  wrote:
>>
>> Hello,
>> I have a simple word count example in Java and I can run this in Eclipse
>> (code at the bottom)
>>
>> I then create a jar file from it and try to run it from the cmd
>>
>>
>> java -jar C:\Users\Owner\Desktop\wordcount.jar Data/testfile.txt
>>
>> But I get this error?
>>
>> I think the main error is:
>> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to
>> find
>> data source: text*
>>
>> Any advise on how to run this jar file in spark would be appreciated
>>
>>
>> Using Spark's default log4j profile:
>> org/apache/spark/log4j-defaults.properties
>> 16/12/07 15:16:41 INFO SparkContext: Running Spark version 2.0.2
>> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls to: Owner
>> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls to: Owner
>> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls groups to:
>> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls groups to:
>> 16/12/07 15:16:42 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users  with view permissions: Set(Owner);
>> groups
>> with view 

Re: Running spark from Eclipse and then Jar

[Iman Mohtashemi]

No but I tried that too and still didn't work. Where are the files being
read from? From the local machine or HDFS? Do I need to get the files to
HDFS first? In Eclipse I just point to the location of the directory?

On Wed, Dec 7, 2016 at 3:34 PM Md. Rezaul Karim <
rezaul.ka...@insight-centre.org> wrote:

> Hi,
>
> You should prepare your jar file (from your Spark application written in
> Java) with all the necessary dependencies. You can create a Maven project
> on Eclipse by specifying the dependencies in a Maven friendly pom.xml file.
>
> For building the jar with the dependencies and *main class (since you are
> getting the **ClassNotFoundException)* your pom.xml should contain the
> following in the *build *tag (example main class is marked in Red color):
>
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-eclipse-plugin
> 2.9
> 
> true
> false
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-compiler-plugin
> 3.5.1
> 
> ${jdk.version}
> ${jdk.version}
> 
> 
> 
> org.apache.maven.plugins
> maven-shade-plugin
> 2.4.3
> 
> true
> 
> 
> 
> 
> org.apache.maven.plugins
> maven-assembly-plugin
> 2.4.1
> 
> 
> 
>
> jar-with-dependencies
> 
> 
> 
> 
>
> com.example.RandomForest.SongPrediction
> 
> 
>
> 
>
> oozie.launcher.mapreduce.job.user.classpath.first
> true
> 
>
> 
> 
> 
> make-assembly
> 
> package
> 
> single
> 
> 
> 
> 
> 
> 
>
>
> An example pom.xml file has been attached for your reference. Feel free to
> reuse it.
>
>
> Regards,
> _
> *Md. Rezaul Karim,* BSc, MSc
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> 
>
> On 7 December 2016 at 23:18, im281  wrote:
>
> Hello,
> I have a simple word count example in Java and I can run this in Eclipse
> (code at the bottom)
>
> I then create a jar file from it and try to run it from the cmd
>
>
> java -jar C:\Users\Owner\Desktop\wordcount.jar Data/testfile.txt
>
> But I get this error?
>
> I think the main error is:
> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to
> find
> data source: text*
>
> Any advise on how to run this jar file in spark would be appreciated
>
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/12/07 15:16:41 INFO SparkContext: Running Spark version 2.0.2
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users  with view permissions: Set(Owner);
> groups
> with view permissions: Set(); users  with modify permissions: Set(Owner);
> groups with modify permissions: Set()
> 16/12/07 15:16:44 INFO Utils: Successfully started service 'sparkDriver' on
> port 10211.
> 16/12/07 15:16:44 INFO SparkEnv: Registering MapOutputTracker
> 16/12/07 15:16:44 INFO SparkEnv: Registering BlockManagerMaster
> 16/12/07 15:16:44 INFO DiskBlockManager: Created local directory at
>
> C:\Users\Owner\AppData\Local\Temp\blockmgr-b4b1960b-08fc-44fd-a75e-1a0450556873
> 16/12/07 15:16:44 INFO MemoryStore: MemoryStore started with capacity
> 1984.5
> MB
> 16/12/07 15:16:45 INFO SparkEnv: Registering OutputCommitCoordinator
> 16/12/07 15:16:45 INFO Utils: Successfully started service 'SparkUI' on
> port
> 4040.
> 16/12/07 15:16:45 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> http://192.168.19.2:4040
> 16/12/07 15:16:45 INFO Executor: Starting executor ID driver on host
> localhost
> 16/12/07 15:16:45 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 10252.
> 16/12/07 

Re: Running spark from Eclipse and then Jar

[Md. Rezaul Karim]

Hi,

You should prepare your jar file (from your Spark application written in
Java) with all the necessary dependencies. You can create a Maven project
on Eclipse by specifying the dependencies in a Maven friendly pom.xml file.

For building the jar with the dependencies and *main class (since you are
getting the **ClassNotFoundException)* your pom.xml should contain the
following in the *build *tag (example main class is marked in Red color):





org.apache.maven.plugins
maven-eclipse-plugin
2.9

true
false




org.apache.maven.plugins
maven-compiler-plugin
3.5.1

${jdk.version}
${jdk.version}



org.apache.maven.plugins
maven-shade-plugin
2.4.3

true




org.apache.maven.plugins
maven-assembly-plugin
2.4.1



jar-with-dependencies





com.example.RandomForest.SongPrediction





oozie.launcher.mapreduce.job.user.classpath.first
true





make-assembly

package

single








An example pom.xml file has been attached for your reference. Feel free to
reuse it.


Regards,
_
*Md. Rezaul Karim,* BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web: http://www.reza-analytics.eu/index.html


On 7 December 2016 at 23:18, im281  wrote:

> Hello,
> I have a simple word count example in Java and I can run this in Eclipse
> (code at the bottom)
>
> I then create a jar file from it and try to run it from the cmd
>
>
> java -jar C:\Users\Owner\Desktop\wordcount.jar Data/testfile.txt
>
> But I get this error?
>
> I think the main error is:
> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to
> find
> data source: text*
>
> Any advise on how to run this jar file in spark would be appreciated
>
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/12/07 15:16:41 INFO SparkContext: Running Spark version 2.0.2
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users  with view permissions: Set(Owner);
> groups
> with view permissions: Set(); users  with modify permissions: Set(Owner);
> groups with modify permissions: Set()
> 16/12/07 15:16:44 INFO Utils: Successfully started service 'sparkDriver' on
> port 10211.
> 16/12/07 15:16:44 INFO SparkEnv: Registering MapOutputTracker
> 16/12/07 15:16:44 INFO SparkEnv: Registering BlockManagerMaster
> 16/12/07 15:16:44 INFO DiskBlockManager: Created local directory at
> C:\Users\Owner\AppData\Local\Temp\blockmgr-b4b1960b-08fc-
> 44fd-a75e-1a0450556873
> 16/12/07 15:16:44 INFO MemoryStore: MemoryStore started with capacity
> 1984.5
> MB
> 16/12/07 15:16:45 INFO SparkEnv: Registering OutputCommitCoordinator
> 16/12/07 15:16:45 INFO Utils: Successfully started service 'SparkUI' on
> port
> 4040.
> 16/12/07 15:16:45 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> http://192.168.19.2:4040
> 16/12/07 15:16:45 INFO Executor: Starting executor ID driver on host
> localhost
> 16/12/07 15:16:45 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 10252.
> 16/12/07 15:16:45 INFO NettyBlockTransferService: Server created on
> 192.168.19.2:10252
> 16/12/07 15:16:45 INFO BlockManagerMaster: Registering BlockManager
> BlockManagerId(driver, 192.168.19.2, 10252)
> 16/12/07 15:16:45 INFO BlockManagerMasterEndpoint: Registering block
> manager
> 192.168.19.2:10252 with 1984.5 MB RAM, BlockManagerId(driver,
> 192.168.19.2,
> 10252)
> 16/12/07 15:16:45 INFO BlockManagerMaster: Registered BlockManager
> BlockManagerId(driver, 

Re: Running spark from Eclipse and then Jar

[Gmail]

Don't you need to provide your class name "JavaWordCount"?

Thanks,
Vasu. 

> On Dec 7, 2016, at 3:18 PM, im281  wrote:
> 
> Hello,
> I have a simple word count example in Java and I can run this in Eclipse
> (code at the bottom)
> 
> I then create a jar file from it and try to run it from the cmd
> 
> 
> java -jar C:\Users\Owner\Desktop\wordcount.jar Data/testfile.txt
> 
> But I get this error?
> 
> I think the main error is:
> *Exception in thread "main" java.lang.ClassNotFoundException: Failed to find
> data source: text*
> 
> Any advise on how to run this jar file in spark would be appreciated
> 
> 
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/12/07 15:16:41 INFO SparkContext: Running Spark version 2.0.2
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls to: Owner
> 16/12/07 15:16:42 INFO SecurityManager: Changing view acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: Changing modify acls groups to:
> 16/12/07 15:16:42 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users  with view permissions: Set(Owner); groups
> with view permissions: Set(); users  with modify permissions: Set(Owner);
> groups with modify permissions: Set()
> 16/12/07 15:16:44 INFO Utils: Successfully started service 'sparkDriver' on
> port 10211.
> 16/12/07 15:16:44 INFO SparkEnv: Registering MapOutputTracker
> 16/12/07 15:16:44 INFO SparkEnv: Registering BlockManagerMaster
> 16/12/07 15:16:44 INFO DiskBlockManager: Created local directory at
> C:\Users\Owner\AppData\Local\Temp\blockmgr-b4b1960b-08fc-44fd-a75e-1a0450556873
> 16/12/07 15:16:44 INFO MemoryStore: MemoryStore started with capacity 1984.5
> MB
> 16/12/07 15:16:45 INFO SparkEnv: Registering OutputCommitCoordinator
> 16/12/07 15:16:45 INFO Utils: Successfully started service 'SparkUI' on port
> 4040.
> 16/12/07 15:16:45 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> http://192.168.19.2:4040
> 16/12/07 15:16:45 INFO Executor: Starting executor ID driver on host
> localhost
> 16/12/07 15:16:45 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 10252.
> 16/12/07 15:16:45 INFO NettyBlockTransferService: Server created on
> 192.168.19.2:10252
> 16/12/07 15:16:45 INFO BlockManagerMaster: Registering BlockManager
> BlockManagerId(driver, 192.168.19.2, 10252)
> 16/12/07 15:16:45 INFO BlockManagerMasterEndpoint: Registering block manager
> 192.168.19.2:10252 with 1984.5 MB RAM, BlockManagerId(driver, 192.168.19.2,
> 10252)
> 16/12/07 15:16:45 INFO BlockManagerMaster: Registered BlockManager
> BlockManagerId(driver, 192.168.19.2, 10252)
> 16/12/07 15:16:46 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> 16/12/07 15:16:46 INFO SharedState: Warehouse path is
> 'file:/C:/Users/Owner/spark-warehouse'.
> Exception in thread "main" java.lang.ClassNotFoundException: Failed to find
> data source: text. Please find packages at
> https://cwiki.apache.org/confluence/display/SPARK/Third+Party+Projects
>at
> org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:148)
>at
> org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:79)
>at
> org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:79)
>at
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
>at
> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
>at
> org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:504)
>at
> org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:540)
>at
> org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:513)
>at JavaWordCount.main(JavaWordCount.java:57)
> Caused by: java.lang.ClassNotFoundException: text.DefaultSource
>at java.net.URLClassLoader.findClass(Unknown Source)
>at java.lang.ClassLoader.loadClass(Unknown Source)
>at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
>at java.lang.ClassLoader.loadClass(Unknown Source)
>at
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132)
>at
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132)
>at scala.util.Try$.apply(Try.scala:192)
>at
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132)
>at
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132)
>at scala.util.Try.orElse(Try.scala:84)
>at
> org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:132)
>