subject:"Re\: Issue while creating spark app"

Re: Issue while creating spark app

2022-02-28 Thread ashok34...@yahoo.com.INVALID

 Thanks for all these useful info
Hi all
What is the current trend. Is it Spark on Scala with intellij or Spark on 
python with pycharm. 
I am curious because I have moderate experience with Spark on both Scala and 
python and want to focus on Scala OR python going forward with the intention of 
joining new companies in Big Data.
thanking you
On Sunday, 27 February 2022, 21:32:05 GMT, Mich Talebzadeh 
 wrote:  
 
 Might as well update the artefacts to the correct versions hopefully. 
Downloaded scala 2.12.8

 scala -version



Scala code runner version 2.12.8 -- Copyright 2002-2018, LAMP/EPFL and 
Lightbend, Inc.

Edited the pom.xml as below
http://maven.apache.org/POM/4.0.0"         
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>    
4.0.0
    spark    MichTest    
1.0
            8        
8        
            org.scala-lang        
scala-library        2.12.8    
            org.apache.spark        
spark-core_2.13        3.2.1    
            org.apache.spark        
spark-sql_2.13        3.2.1    
    
and built with maven. All good




   view my Linkedin profile




 https://en.everybodywiki.com/Mich_Talebzadeh

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
damage or destructionof data or any other property which may arise from relying 
on this email's technical content is explicitly disclaimed.The author will in 
no case be liable for any monetary damages arising from suchloss, damage or 
destruction. 

 


On Sun, 27 Feb 2022 at 20:16, Mich Talebzadeh  wrote:

Thanks Bjorn. I am aware of that. I  just really wanted to create the uber jar 
files with both sbt and maven in Intellij.




cheers







   view my Linkedin profile




 https://en.everybodywiki.com/Mich_Talebzadeh

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
damage or destructionof data or any other property which may arise from relying 
on this email's technical content is explicitly disclaimed.The author will in 
no case be liable for any monetary damages arising from suchloss, damage or 
destruction. 

 


On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen  wrote:

Mitch: You are using scala 2.11 to do this. Have a look at Building Spark 
"Spark requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark 
3.0.0."
søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh :

OK I decided to give a try to maven.
Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip 
apache-maven-3.8.4-bin.zip
Then added to Windows env variable as MVN_HOME and added the bin directory to 
path in windows. Restart intellij to pick up the correct path.
Again on the command line in intellij do
mvn -vApache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)Maven home: 
d:\temp\apache-maven-3.8.4Java version: 1.8.0_73, vendor: Oracle Corporation, 
runtime: C:\Program Files\Java\jdk1.8.0_73\jreDefault locale: en_GB, platform 
encoding: Cp1252OS name: "windows 10", version: "10.0", arch: "amd64", family: 
"windows"
in Intellij add maven support to your project. Follow this link Add Maven 
support to an existing project
There will be a pom.xml file under project directory 


Edit that pom.xml file and add the following
http://maven.apache.org/POM/4.0.0";
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
4.0.0

spark
MichTest
1.0


8
8



org.scala-lang
scala-library
2.11.7


org.apache.spark
spark-core_2.10
2.0.0


org.apache.spark
spark-sql_2.10
2.0.0



In intellij open a Terminal under project sub-directory where the pom file is 
created and you edited.




 mvn clean

[INFO] Scanning for projects...

[INFO] 

[INFO] ---< spark:MichTest >---

[INFO] Building MichTest 1.0

[INFO] [ jar ]-

[INFO] 

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---

[INFO] Deleting D:\temp\intellij\MichTest\target

[INFO] 

[INFO] BUILD SUCCESS

[INFO] 

[INFO] Total time:  4.451 s

[INFO] Finished at: 2022-02-27T19:37:57Z



[INFO] 


mvn compile

[INFO] Scanning for projects...

[INFO] 

[INFO] ---< spark:MichTest >---

[INFO] Building MichTest 1.0

[INFO] [ jar ]-

[INFO] 

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ MichTest

Re: Issue while creating spark app

2022-02-27 Thread Mich Talebzadeh

Might as well update the artefacts to the correct versions hopefully.
Downloaded scala 2.12.8

 scala -version

Scala code runner version 2.12.8 -- Copyright 2002-2018, LAMP/EPFL and
Lightbend, Inc.

Edited the pom.xml as below


http://maven.apache.org/POM/4.0.0";
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
4.0.0

spark
MichTest
1.0


8
8



org.scala-lang
scala-library
*2.12.8*


org.apache.spark
*spark-core_2.13*
*3.2.1*


org.apache.spark
*spark-sql_2.13*
*3.2.1*




and built with maven. All good


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sun, 27 Feb 2022 at 20:16, Mich Talebzadeh 
wrote:

> Thanks Bjorn. I am aware of that. I  just really wanted to create the uber
> jar files with both sbt and maven in Intellij.
>
>
> cheers
>
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen 
> wrote:
>
>> Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
>>  "Spark
>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
>> 3.0.0."
>>
>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
>> mich.talebza...@gmail.com>:
>>
>>> OK I decided to give a try to maven.
>>>
>>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>>> apache-maven-3.8.4-bin.zip
>>>
>>> Then added to Windows env variable as MVN_HOME and added the bin
>>> directory to path in windows. Restart intellij to pick up the correct path.
>>>
>>> Again on the command line in intellij do
>>>
>>> *mvn -v*
>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>> Maven home: d:\temp\apache-maven-3.8.4
>>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
>>> Files\Java\jdk1.8.0_73\jre
>>> Default locale: en_GB, platform encoding: Cp1252
>>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>>>
>>> in Intellij add maven support to your project. Follow this link Add
>>> Maven support to an existing project
>>> 
>>>
>>> There will be a pom.xml file under project directory
>>>
>>> [image: image.png]
>>>
>>> Edit that pom.xml file and add the following
>>>
>>> 
>>> http://maven.apache.org/POM/4.0.0";
>>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
>>> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>>> 4.0.0
>>>
>>> spark
>>> MichTest
>>> 1.0
>>>
>>> 
>>> 8
>>> 8
>>> 
>>> 
>>> 
>>> org.scala-lang
>>> scala-library
>>> 2.11.7
>>> 
>>> 
>>> org.apache.spark
>>> spark-core_2.10
>>> 2.0.0
>>> 
>>> 
>>> org.apache.spark
>>> spark-sql_2.10
>>> 2.0.0
>>> 
>>> 
>>> 
>>>
>>> In intellij open a Terminal under project sub-directory where the pom
>>> file is created and you edited.
>>>
>>>
>>>  *mvn clean*
>>>
>>> [INFO] Scanning for projects...
>>>
>>> [INFO]
>>>
>>> [INFO] ---< spark:MichTest
>>> >---
>>>
>>> [INFO] Building MichTest 1.0
>>>
>>> [INFO] [ jar
>>> ]-
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>>
>>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>>
>>> [INFO]
>>> 
>>>
>>> [INFO] BUILD SUCCESS
>>>
>>> [INFO]
>>> 
>>>
>>> [INFO] Total time:  4.451 s
>>>
>>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>>
>>> [INFO]
>>> ---

Re: Issue while creating spark app

2022-02-27 Thread Bjørn Jørgensen

Rajat Kumer wrote: "Cannot find project Scala library 2.12.12 for module
SparkSimpleApp"

So when I google this error message I find scala project maven sync failed




søn. 27. feb. 2022 kl. 21:59 skrev Mich Talebzadeh <
mich.talebza...@gmail.com>:

> sorry which error?
>
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 27 Feb 2022 at 20:52, Bjørn Jørgensen 
> wrote:
>
>> Anyway I did google on your error and found this scala project maven
>> sync failed 
>> Is this the same as the one you are getting?
>>
>>
>> søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh <
>> mich.talebza...@gmail.com>:
>>
>>> Thanks Bjorn. I am aware of that. I  just really wanted to create the
>>> uber jar files with both sbt and maven in Intellij.
>>>
>>>
>>> cheers
>>>
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen 
>>> wrote:
>>>
 Mitch: You are using scala 2.11 to do this. Have a look at Building
 Spark  "Spark
 requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
 3.0.0."

 søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
 mich.talebza...@gmail.com>:

> OK I decided to give a try to maven.
>
> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
> apache-maven-3.8.4-bin.zip
>
> Then added to Windows env variable as MVN_HOME and added the bin
> directory to path in windows. Restart intellij to pick up the correct 
> path.
>
> Again on the command line in intellij do
>
> *mvn -v*
> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
> Maven home: d:\temp\apache-maven-3.8.4
> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime:
> C:\Program Files\Java\jdk1.8.0_73\jre
> Default locale: en_GB, platform encoding: Cp1252
> OS name: "windows 10", version: "10.0", arch: "amd64", family:
> "windows"
>
> in Intellij add maven support to your project. Follow this link Add
> Maven support to an existing project
> 
>
> There will be a pom.xml file under project directory
>
> [image: image.png]
>
> Edit that pom.xml file and add the following
>
> 
> http://maven.apache.org/POM/4.0.0";
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> 4.0.0
>
> spark
> MichTest
> 1.0
>
> 
> 8
> 8
> 
> 
> 
> org.scala-lang
> scala-library
> 2.11.7
> 
> 
> org.apache.spark
> spark-core_2.10
> 2.0.0
> 
> 
> org.apache.spark
> spark-sql_2.10
> 2.0.0
> 
> 
> 
>
> In intellij open a Terminal under project sub-directory where the pom
> file is created and you edited.
>
>
>  *mvn clean*
>
> [INFO] Scanning for projects...
>
> [INFO]
>
> [INFO] ---< spark:MichTest
> >---
>
> [INFO] Building MichTest 1.0
>
> [INFO] [ jar
> ]-
>
> [INFO]
>
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>
> [INFO] Deleting D:\temp\intellij\MichTest\target
>
> [INFO]
> 
>
> [INFO] BUILD SUCCESS
>
> [INFO]
> 
>
> [INFO] Total time:  4.451 s
>
>

Re: Issue while creating spark app

2022-02-27 Thread Mich Talebzadeh

sorry which error?



   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sun, 27 Feb 2022 at 20:52, Bjørn Jørgensen 
wrote:

> Anyway I did google on your error and found this scala project maven sync
> failed 
> Is this the same as the one you are getting?
>
>
> søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh <
> mich.talebza...@gmail.com>:
>
>> Thanks Bjorn. I am aware of that. I  just really wanted to create the
>> uber jar files with both sbt and maven in Intellij.
>>
>>
>> cheers
>>
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen 
>> wrote:
>>
>>> Mitch: You are using scala 2.11 to do this. Have a look at Building
>>> Spark  "Spark
>>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
>>> 3.0.0."
>>>
>>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
>>> mich.talebza...@gmail.com>:
>>>
 OK I decided to give a try to maven.

 Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
 apache-maven-3.8.4-bin.zip

 Then added to Windows env variable as MVN_HOME and added the bin
 directory to path in windows. Restart intellij to pick up the correct path.

 Again on the command line in intellij do

 *mvn -v*
 Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
 Maven home: d:\temp\apache-maven-3.8.4
 Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
 Files\Java\jdk1.8.0_73\jre
 Default locale: en_GB, platform encoding: Cp1252
 OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

 in Intellij add maven support to your project. Follow this link Add
 Maven support to an existing project
 

 There will be a pom.xml file under project directory

 [image: image.png]

 Edit that pom.xml file and add the following

 
 http://maven.apache.org/POM/4.0.0";
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
 http://maven.apache.org/xsd/maven-4.0.0.xsd";>
 4.0.0

 spark
 MichTest
 1.0

 
 8
 8
 
 
 
 org.scala-lang
 scala-library
 2.11.7
 
 
 org.apache.spark
 spark-core_2.10
 2.0.0
 
 
 org.apache.spark
 spark-sql_2.10
 2.0.0
 
 
 

 In intellij open a Terminal under project sub-directory where the pom
 file is created and you edited.


  *mvn clean*

 [INFO] Scanning for projects...

 [INFO]

 [INFO] ---< spark:MichTest
 >---

 [INFO] Building MichTest 1.0

 [INFO] [ jar
 ]-

 [INFO]

 [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---

 [INFO] Deleting D:\temp\intellij\MichTest\target

 [INFO]
 

 [INFO] BUILD SUCCESS

 [INFO]
 

 [INFO] Total time:  4.451 s

 [INFO] Finished at: 2022-02-27T19:37:57Z

 [INFO]
 

 *mvn compile*

 [INFO] Scanning for projects...

 [INFO]

 [INFO] ---< spark:MichTest
 >---

 [INFO] Building MichTest 1.0

 [INFO] [ jar
 ]-

 [INFO]

 [INFO] --- m

Re: Issue while creating spark app

2022-02-27 Thread Bjørn Jørgensen

Anyway I did google on your error and found this scala project maven sync
failed 
Is this the same as the one you are getting?


søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh <
mich.talebza...@gmail.com>:

> Thanks Bjorn. I am aware of that. I  just really wanted to create the uber
> jar files with both sbt and maven in Intellij.
>
>
> cheers
>
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen 
> wrote:
>
>> Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
>>  "Spark
>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
>> 3.0.0."
>>
>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
>> mich.talebza...@gmail.com>:
>>
>>> OK I decided to give a try to maven.
>>>
>>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>>> apache-maven-3.8.4-bin.zip
>>>
>>> Then added to Windows env variable as MVN_HOME and added the bin
>>> directory to path in windows. Restart intellij to pick up the correct path.
>>>
>>> Again on the command line in intellij do
>>>
>>> *mvn -v*
>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>> Maven home: d:\temp\apache-maven-3.8.4
>>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
>>> Files\Java\jdk1.8.0_73\jre
>>> Default locale: en_GB, platform encoding: Cp1252
>>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>>>
>>> in Intellij add maven support to your project. Follow this link Add
>>> Maven support to an existing project
>>> 
>>>
>>> There will be a pom.xml file under project directory
>>>
>>> [image: image.png]
>>>
>>> Edit that pom.xml file and add the following
>>>
>>> 
>>> http://maven.apache.org/POM/4.0.0";
>>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
>>> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>>> 4.0.0
>>>
>>> spark
>>> MichTest
>>> 1.0
>>>
>>> 
>>> 8
>>> 8
>>> 
>>> 
>>> 
>>> org.scala-lang
>>> scala-library
>>> 2.11.7
>>> 
>>> 
>>> org.apache.spark
>>> spark-core_2.10
>>> 2.0.0
>>> 
>>> 
>>> org.apache.spark
>>> spark-sql_2.10
>>> 2.0.0
>>> 
>>> 
>>> 
>>>
>>> In intellij open a Terminal under project sub-directory where the pom
>>> file is created and you edited.
>>>
>>>
>>>  *mvn clean*
>>>
>>> [INFO] Scanning for projects...
>>>
>>> [INFO]
>>>
>>> [INFO] ---< spark:MichTest
>>> >---
>>>
>>> [INFO] Building MichTest 1.0
>>>
>>> [INFO] [ jar
>>> ]-
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>>
>>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>>
>>> [INFO]
>>> 
>>>
>>> [INFO] BUILD SUCCESS
>>>
>>> [INFO]
>>> 
>>>
>>> [INFO] Total time:  4.451 s
>>>
>>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>>
>>> [INFO]
>>> 
>>>
>>> *mvn compile*
>>>
>>> [INFO] Scanning for projects...
>>>
>>> [INFO]
>>>
>>> [INFO] ---< spark:MichTest
>>> >---
>>>
>>> [INFO] Building MichTest 1.0
>>>
>>> [INFO] [ jar
>>> ]-
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>> MichTest ---
>>>
>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>> resources, i.e. build is platform dependent!
>>>
>>> [INFO] Copying 0 resource
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>> MichTest ---
>>>
>>> [INFO] Nothing to compile - all classes are up to date
>>>
>>> [INFO]
>>> 
>>>
>>> [INFO] BUILD SUCCESS
>>>
>>> [INFO]
>>> 
>>>
>>> [INFO] Total time:  1.242 s
>>>
>>> [INFO] Finished at: 2022-02-27T19:38:58Z
>>>
>>> [INFO]

Re: Issue while creating spark app

2022-02-27 Thread Mich Talebzadeh

Thanks Bjorn. I am aware of that. I  just really wanted to create the uber
jar files with both sbt and maven in Intellij.


cheers



   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen 
wrote:

> Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
>  "Spark
> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
> 3.0.0."
>
> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
> mich.talebza...@gmail.com>:
>
>> OK I decided to give a try to maven.
>>
>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>> apache-maven-3.8.4-bin.zip
>>
>> Then added to Windows env variable as MVN_HOME and added the bin
>> directory to path in windows. Restart intellij to pick up the correct path.
>>
>> Again on the command line in intellij do
>>
>> *mvn -v*
>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>> Maven home: d:\temp\apache-maven-3.8.4
>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
>> Files\Java\jdk1.8.0_73\jre
>> Default locale: en_GB, platform encoding: Cp1252
>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>>
>> in Intellij add maven support to your project. Follow this link Add
>> Maven support to an existing project
>> 
>>
>> There will be a pom.xml file under project directory
>>
>> [image: image.png]
>>
>> Edit that pom.xml file and add the following
>>
>> 
>> http://maven.apache.org/POM/4.0.0";
>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
>> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>> 4.0.0
>>
>> spark
>> MichTest
>> 1.0
>>
>> 
>> 8
>> 8
>> 
>> 
>> 
>> org.scala-lang
>> scala-library
>> 2.11.7
>> 
>> 
>> org.apache.spark
>> spark-core_2.10
>> 2.0.0
>> 
>> 
>> org.apache.spark
>> spark-sql_2.10
>> 2.0.0
>> 
>> 
>> 
>>
>> In intellij open a Terminal under project sub-directory where the pom
>> file is created and you edited.
>>
>>
>>  *mvn clean*
>>
>> [INFO] Scanning for projects...
>>
>> [INFO]
>>
>> [INFO] ---< spark:MichTest
>> >---
>>
>> [INFO] Building MichTest 1.0
>>
>> [INFO] [ jar
>> ]-
>>
>> [INFO]
>>
>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>
>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>
>> [INFO]
>> 
>>
>> [INFO] BUILD SUCCESS
>>
>> [INFO]
>> 
>>
>> [INFO] Total time:  4.451 s
>>
>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>
>> [INFO]
>> 
>>
>> *mvn compile*
>>
>> [INFO] Scanning for projects...
>>
>> [INFO]
>>
>> [INFO] ---< spark:MichTest
>> >---
>>
>> [INFO] Building MichTest 1.0
>>
>> [INFO] [ jar
>> ]-
>>
>> [INFO]
>>
>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>> MichTest ---
>>
>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>> resources, i.e. build is platform dependent!
>>
>> [INFO] Copying 0 resource
>>
>> [INFO]
>>
>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
>> ---
>>
>> [INFO] Nothing to compile - all classes are up to date
>>
>> [INFO]
>> 
>>
>> [INFO] BUILD SUCCESS
>>
>> [INFO]
>> 
>>
>> [INFO] Total time:  1.242 s
>>
>> [INFO] Finished at: 2022-02-27T19:38:58Z
>>
>> [INFO]
>> 
>>
>> Now create the package
>>
>>  *mvn package*
>> [INFO] Scanning for projects...
>> [INFO]
>> [INFO] ---< spark:MichTest
>> >---
>> [INFO] Building MichTest 1.0
>> [INFO] [ jar
>> ]-
>> [INFO]
>> [INFO] --- maven-resources-plugin:2.6:resources (default-resource

Re: Issue while creating spark app

2022-02-27 Thread Bjørn Jørgensen

Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
 "Spark requires
Scala 2.12/2.13; support for Scala 2.11 was removed in Spark 3.0.0."

søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
mich.talebza...@gmail.com>:

> OK I decided to give a try to maven.
>
> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
> apache-maven-3.8.4-bin.zip
>
> Then added to Windows env variable as MVN_HOME and added the bin directory
> to path in windows. Restart intellij to pick up the correct path.
>
> Again on the command line in intellij do
>
> *mvn -v*
> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
> Maven home: d:\temp\apache-maven-3.8.4
> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
> Files\Java\jdk1.8.0_73\jre
> Default locale: en_GB, platform encoding: Cp1252
> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>
> in Intellij add maven support to your project. Follow this link Add Maven
> support to an existing project
> 
>
> There will be a pom.xml file under project directory
>
> [image: image.png]
>
> Edit that pom.xml file and add the following
>
> 
> http://maven.apache.org/POM/4.0.0";
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> 4.0.0
>
> spark
> MichTest
> 1.0
>
> 
> 8
> 8
> 
> 
> 
> org.scala-lang
> scala-library
> 2.11.7
> 
> 
> org.apache.spark
> spark-core_2.10
> 2.0.0
> 
> 
> org.apache.spark
> spark-sql_2.10
> 2.0.0
> 
> 
> 
>
> In intellij open a Terminal under project sub-directory where the pom file
> is created and you edited.
>
>
>  *mvn clean*
>
> [INFO] Scanning for projects...
>
> [INFO]
>
> [INFO] ---< spark:MichTest
> >---
>
> [INFO] Building MichTest 1.0
>
> [INFO] [ jar
> ]-
>
> [INFO]
>
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>
> [INFO] Deleting D:\temp\intellij\MichTest\target
>
> [INFO]
> 
>
> [INFO] BUILD SUCCESS
>
> [INFO]
> 
>
> [INFO] Total time:  4.451 s
>
> [INFO] Finished at: 2022-02-27T19:37:57Z
>
> [INFO]
> 
>
> *mvn compile*
>
> [INFO] Scanning for projects...
>
> [INFO]
>
> [INFO] ---< spark:MichTest
> >---
>
> [INFO] Building MichTest 1.0
>
> [INFO] [ jar
> ]-
>
> [INFO]
>
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> MichTest ---
>
> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
> resources, i.e. build is platform dependent!
>
> [INFO] Copying 0 resource
>
> [INFO]
>
> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
> ---
>
> [INFO] Nothing to compile - all classes are up to date
>
> [INFO]
> 
>
> [INFO] BUILD SUCCESS
>
> [INFO]
> 
>
> [INFO] Total time:  1.242 s
>
> [INFO] Finished at: 2022-02-27T19:38:58Z
>
> [INFO]
> 
>
> Now create the package
>
>  *mvn package*
> [INFO] Scanning for projects...
> [INFO]
> [INFO] ---< spark:MichTest
> >---
> [INFO] Building MichTest 1.0
> [INFO] [ jar
> ]-
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> MichTest ---
> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
> resources, i.e. build is platform dependent!
> [INFO] Copying 0 resource
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
> ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:testResources
> (default-testResources) @ MichTest ---
> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory
> D:\temp\intellij\MichTest\src\test\resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
> MichTest ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- maven-surefire-plugin:2.12.4

Re: Issue while creating spark app

2022-02-27 Thread Mich Talebzadeh

OK I decided to give a try to maven.

Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
apache-maven-3.8.4-bin.zip

Then added to Windows env variable as MVN_HOME and added the bin directory
to path in windows. Restart intellij to pick up the correct path.

Again on the command line in intellij do

*mvn -v*
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: d:\temp\apache-maven-3.8.4
Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
Files\Java\jdk1.8.0_73\jre
Default locale: en_GB, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

in Intellij add maven support to your project. Follow this link Add Maven
support to an existing project


There will be a pom.xml file under project directory

[image: image.png]

Edit that pom.xml file and add the following


http://maven.apache.org/POM/4.0.0";
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
4.0.0

spark
MichTest
1.0


8
8



org.scala-lang
scala-library
2.11.7


org.apache.spark
spark-core_2.10
2.0.0


org.apache.spark
spark-sql_2.10
2.0.0




In intellij open a Terminal under project sub-directory where the pom file
is created and you edited.


 *mvn clean*

[INFO] Scanning for projects...

[INFO]

[INFO] ---< spark:MichTest
>---

[INFO] Building MichTest 1.0

[INFO] [ jar
]-

[INFO]

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---

[INFO] Deleting D:\temp\intellij\MichTest\target

[INFO]


[INFO] BUILD SUCCESS

[INFO]


[INFO] Total time:  4.451 s

[INFO] Finished at: 2022-02-27T19:37:57Z

[INFO]


*mvn compile*

[INFO] Scanning for projects...

[INFO]

[INFO] ---< spark:MichTest
>---

[INFO] Building MichTest 1.0

[INFO] [ jar
]-

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
MichTest ---

[WARNING] Using platform encoding (Cp1252 actually) to copy filtered
resources, i.e. build is platform dependent!

[INFO] Copying 0 resource

[INFO]

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
---

[INFO] Nothing to compile - all classes are up to date

[INFO]


[INFO] BUILD SUCCESS

[INFO]


[INFO] Total time:  1.242 s

[INFO] Finished at: 2022-02-27T19:38:58Z

[INFO]


Now create the package

 *mvn package*
[INFO] Scanning for projects...
[INFO]
[INFO] ---< spark:MichTest
>---
[INFO] Building MichTest 1.0
[INFO] [ jar
]-
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
MichTest ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources)
@ MichTest ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
D:\temp\intellij\MichTest\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
MichTest ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
[INFO] *Building jar:
D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
[INFO]

[INFO] BUILD SUCCESS
[INFO]

[INFO] Total time:  1.511 s
[INFO] Finished at: 2022-02-27T19:40:22Z
[INFO]


sbt should work from the same directory as well.

I find it easier

Re: Issue while creating spark app

2022-02-27 Thread Mich Talebzadeh

Got curious with this intellij stuff.

I recall using sbt rather than MVN so go to terminal in your intellij and
verify what is installed

 sbt -version
sbt version in this project: 1.3.4
sbt script version: 1.3.4

 scala -version

Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL

java --version
openjdk 11.0.7 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)

For now in the directory where you have Main.scala create build.sbt file

// The simplest possible sbt build file is just one line:

scalaVersion := "2.11.7"
// That is, to create a valid sbt build, all you've got to do is define the
// version of Scala you'd like your project to use.

/ To learn more about multi-project builds, head over to the official sbt
// documentation at http://www.scala-sbt.org/documentation.html
libraryDependencies += "org.scala-lang.modules" %%
"scala-parser-combinators" % "1.1.2"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"


This is my Main.scala file a copy from sparkbyexample



package org.example
import org.apache.spark.sql.SparkSession
object SparkSessionTest extends App{
  val spark = SparkSession.builder()
.master("local[1]")
.appName("SparkByExample")
.getOrCreate();

  println("First SparkContext:")
  println("APP Name :"+spark.sparkContext.appName);
  println("Deploy Mode :"+spark.sparkContext.deployMode);
  println("Master :"+spark.sparkContext.master);

  val sparkSession2 = SparkSession.builder()
.master("local[1]")
.appName("SparkByExample-test")
.getOrCreate();

  println("Second SparkContext:")
  println("APP Name :"+sparkSession2.sparkContext.appName);
  println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
  println("Master :"+sparkSession2.sparkContext.master);
}

 Go back to Terminal under directory where you have both files built.sbt
and Main.scala


*sbt clean*

[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins

[info] Loading project definition from
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project

[info] Loading settings for project scala from build.sbt ...

[info] Set current project to MichTest (in build
file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)

[success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM

*sbt compile*
[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
[info] Loading project definition from
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
[info] Loading settings for project scala from build.sbt ...
[info] Set current project to MichTest (in build
file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
[info] Executing in batch mode. For better performance use sbt's shell
[warn] There may be incompatibilities among your library dependencies; run
'evicted' to see detailed eviction warnings.
[info] Compiling 1 Scala source to
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
...
[success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM

 *sbt package*

[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins

[info] Loading project definition from
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project

[info] Loading settings for project scala from build.sbt ...

[info] Set current project to MichTest (in build
file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)

[success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM

 *ls*


Directory:
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala


Mode LastWriteTime Length Name

 - -- 

d- 2/27/2022   7:04 AMnull

d- 2/27/2022   8:33 AMproject

d- 2/27/2022   9:08 AMspark-warehouse

d- 2/27/2022   9:55 AMtarget

-a 2/27/2022   9:17 AM   3511 build.sbt

Note that you have target directory and underneath scala-2.11 (in my case)
and the uber jar file michtest_2.11-1.0.jar

*ls*
Directory:
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
Mode LastWriteTime Length Name
 - -- 
d- 2/27/2022   9:55 AMclasses
d- 2/27/2022   9:55 AMupdate
-a 2/27/2022   9:56 AM   3938 *michtest_2.11-1.0.jar*


These are old stuff but still shows how to create a jar file with sbt

HTH


  view my Linkedin profile




 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction

Re: Issue while creating spark app

2022-02-26 Thread Sean Owen

I don't think any of that is related, no.
How are you dependencies set up? manually with IJ, or in a build file
(Maven, Gradle)? Normally you do the latter and dependencies are taken care
of for you, but you app would definitely have to express a dependency on
Scala libs.

On Sat, Feb 26, 2022 at 4:25 PM Bitfox  wrote:

> Java SDK installed?
>
> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka 
> wrote:
>
>> Hello ,
>>
>> Thanks for replying. I have installed Scala plugin in IntelliJ  first
>> then also it's giving same error
>>
>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>
>> Thanks
>> Rajat
>>
>> On Sun, Feb 27, 2022, 00:52 Bitfox  wrote:
>>
>>> You need to install scala first, the current version for spark is 2.12.15
>>> I would suggest you install scala by sdk which works great.
>>>
>>> Thanks
>>>
>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar 
>>> wrote:
>>>
 Hello Users,

 I am trying to create spark application using Scala(Intellij).
 I have installed Scala plugin in intelliJ still getting below error:-

 Cannot find project Scala library 2.12.12 for module SparkSimpleApp

 Could anyone please help what I am doing wrong?

 Thanks

 Rajat

>>>

Re: Issue while creating spark app

2022-02-26 Thread Bitfox

Java SDK installed?

On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka 
wrote:

> Hello ,
>
> Thanks for replying. I have installed Scala plugin in IntelliJ  first then
> also it's giving same error
>
> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>
> Thanks
> Rajat
>
> On Sun, Feb 27, 2022, 00:52 Bitfox  wrote:
>
>> You need to install scala first, the current version for spark is 2.12.15
>> I would suggest you install scala by sdk which works great.
>>
>> Thanks
>>
>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar 
>> wrote:
>>
>>> Hello Users,
>>>
>>> I am trying to create spark application using Scala(Intellij).
>>> I have installed Scala plugin in intelliJ still getting below error:-
>>>
>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>
>>>
>>> Could anyone please help what I am doing wrong?
>>>
>>> Thanks
>>>
>>> Rajat
>>>
>>

Re: Issue while creating spark app

2022-02-26 Thread Sachit Murarka

Hello ,

Thanks for replying. I have installed Scala plugin in IntelliJ  first then
also it's giving same error

Cannot find project Scala library 2.12.12 for module SparkSimpleApp

Thanks
Rajat

On Sun, Feb 27, 2022, 00:52 Bitfox  wrote:

> You need to install scala first, the current version for spark is 2.12.15
> I would suggest you install scala by sdk which works great.
>
> Thanks
>
> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar 
> wrote:
>
>> Hello Users,
>>
>> I am trying to create spark application using Scala(Intellij).
>> I have installed Scala plugin in intelliJ still getting below error:-
>>
>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>
>>
>> Could anyone please help what I am doing wrong?
>>
>> Thanks
>>
>> Rajat
>>
>

Re: Issue while creating spark app

2022-02-26 Thread Bitfox

You need to install scala first, the current version for spark is 2.12.15
I would suggest you install scala by sdk which works great.

Thanks

On Sun, Feb 27, 2022 at 12:10 AM rajat kumar 
wrote:

> Hello Users,
>
> I am trying to create spark application using Scala(Intellij).
> I have installed Scala plugin in intelliJ still getting below error:-
>
> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>
>
> Could anyone please help what I am doing wrong?
>
> Thanks
>
> Rajat
>

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

Re: Issue while creating spark app

13 matches

Site Navigation

Mail list logo

Footer information