Change
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.13.11-M2</version>
</dependency>
<dependency>
to
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <lingzhe....@hirain.com>:
> Hi Mich,
>
> Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8
> <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>.
> Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to
> the major(13) and the minor version(8). Not using any of these may cause
> unexpected behaviour(though scala claims compatibility among minor version
> changes, I've encountered problem using the scala package with the same
> major version and different minor version. That may due to bug fixes and
> upgrade of scala itself.).
> And although I did not encountered such problem, this
> <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for you.
>
> ------------------------------
> Best Regards!
> ...........................................................................
> Lingzhe Sun
> Hirain Technology
>
>
> *From:* Mich Talebzadeh <mich.talebza...@gmail.com>
> *Date:* 2023-05-29 17:55
> *To:* Bjørn Jørgensen <bjornjorgen...@gmail.com>
> *CC:* user @spark <user@spark.apache.org>
> *Subject:* Re: maven with Spark 3.4.0 fails compilation
> Thanks for your helpful comments Bjorn.
>
> I managed to compile the code with maven but when it run it fails with
>
> Application is ReduceByKey
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.package$.Seq()Lscala/collection/immutable/Seq$;
> at ReduceByKey$.main(ReduceByKey.scala:23)
> at ReduceByKey.main(ReduceByKey.scala)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
> at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
> at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> I attach the pom.xml and the sample scala code is self contained and
> basic. Again it runs with SBT with no issues.
>
> FYI, my scala version on host is
>
> scala -version
> Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and
> Lightbend, Inc.
>
> I think I have a scala incompatible somewhere again
>
> Cheers
>
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bjornjorgen...@gmail.com>
> wrote:
>
>> From chatgpt4
>>
>>
>> The problem appears to be that there is a mismatch between the version of
>> Scala used by the Scala Maven plugin and the version of the Scala library
>> defined as a dependency in your POM. You've defined your Scala version in
>> your properties as `2.12.17` but you're pulling in `scala-library` version
>> `2.13.6` as a dependency.
>>
>> The Scala Maven plugin will be using the Scala version defined in the
>> `scala.version` property for compilation, but then it tries to load classes
>> from a different Scala version, hence the error.
>>
>> To resolve this issue, make sure the `scala.version` property matches the
>> version of `scala-library` defined in your dependencies. In your case, you
>> may want to change `scala.version` to `2.13.6`.
>>
>> Here's the corrected part of your POM:
>>
>> ```xml
>> <properties>
>> <maven.compiler.source>1.7</maven.compiler.source>
>> <maven.compiler.target>1.7</maven.compiler.target>
>> <encoding>UTF-8</encoding>
>> <scala.version>2.13.6</scala.version> <!-- Here's the change -->
>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>> </properties>
>> ```
>>
>> Additionally, ensure that the Scala versions in the Spark dependencies
>> match the `scala.version` property as well. If you've updated the Scala
>> version to `2.13.6`, the artifactIds for Spark dependencies should be
>> `spark-core_2.13` and `spark-sql_2.13`.
>>
>> Another thing to consider: your Java version defined in
>> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is
>> quite outdated and might not be compatible with the latest versions of
>> these libraries. Consider updating to a more recent version of Java, such
>> as Java 8 or above, depending on the requirements of the libraries you're
>> using.
>>
>>
>>
>> The same problem persists in this updated POM file - there's a mismatch
>> in the Scala version declared in the properties and the version used in
>> your dependencies. Here's what you need to update:
>>
>> 1. Update the Scala version in your properties to match the Scala library
>> and your Spark dependencies:
>>
>> ```xml
>> <properties>
>> <maven.compiler.source>1.7</maven.compiler.source>
>> <maven.compiler.target>1.7</maven.compiler.target>
>> <encoding>UTF-8</encoding>
>> <scala.version>2.13.6</scala.version> <!-- updated Scala version -->
>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>> </properties>
>> ```
>>
>> 2. Make sure all your Spark dependencies use the same Scala version. In
>> this case, I see `spark-streaming-kafka_2.11` which should be
>> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`.
>>
>> ```xml
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to
>> 2.13 -->
>> <version>1.6.3</version> <!-- this might also need to be updated as
>> this is a very old version -->
>> <scope>provided</scope>
>> </dependency>
>> ```
>>
>> 3. As mentioned in the previous message, your Java version
>> (`maven.compiler.source` and `maven.compiler.target`) is also quite
>> outdated. Depending on the requirements of the libraries you're using, you
>> might want to update this to a newer version, such as Java 8 or above.
>>
>> Finally, ensure that the correct versions of these libraries exist in
>> your Maven repository or are available in the central Maven repository. If
>> the versions don't match, Maven will not be able to find and download the
>> correct dependencies, which can lead to problems.
>>
>> Please note that it's crucial to maintain consistency in your Scala and
>> Java versions across your project and its dependencies to avoid these kinds
>> of issues. Mixing versions can lead to binary incompatibility errors, such
>> as the one you're seeing.
>>
>>
>>
>>
>>
>>
>>
>> The differences in behavior between SBT and Maven when resolving Scala
>> dependencies can be attributed to how they each handle Scala binary
>> versions.
>>
>> SBT is specifically designed for Scala projects and has built-in support
>> for handling Scala's binary compatibility issues. When you define a Scala
>> library dependency in SBT, you can specify the Scala binary version as
>> "_2.12", "_2.13", etc. in the library artifact name. SBT will then
>> automatically replace this with the actual Scala binary version defined in
>> your SBT configuration. So, if you've defined different Scala versions for
>> your project and for a specific dependency, SBT will handle this gracefully.
>>
>> Maven, on the other hand, is more generic and does not have the same
>> level of built-in support for Scala's binary versions. In Maven, the Scala
>> version is typically hardcoded in the artifact name, and if this doesn't
>> match the actual Scala version used in your project, it can lead to binary
>> compatibility issues.
>>
>> This difference between SBT and Maven means that a project can work fine
>> when built with SBT but fail when built with Maven, due to these Scala
>> version discrepancies. To avoid these issues in Maven, you need to ensure
>> that the Scala version is consistent across your project configuration and
>> all your dependencies.
>>
>> Also, another reason for this can be because SBT often fetches
>> dependencies on the fly, while Maven requires them to be explicitly
>> declared. SBT can download and link the correct Scala library version even
>> if it's not explicitly declared, while Maven will only use the versions
>> that have been specified in the POM file.
>>
>> To summarize, SBT is more flexible and Scala-oriented than Maven, which
>> can lead to different behavior when handling Scala dependencies. When using
>> Maven for Scala projects, it's essential to ensure that all Scala versions
>> match across the project.
>>
>> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh <
>> mich.talebza...@gmail.com>:
>>
>>> This compilation works fine with SBT but fails with maven!
>>>
>>> Spark version 3.4.0
>>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
>>> Java version: 11.0.1, vendor: Oracle Corporation, runtime:
>>> /opt/jdk-11.0.1
>>>
>>> This from the pom.xml file
>>>
>>> <project xmlns="https://maven.apache.org/POM/4.0.0"; xmlns:xsi="
>>> https://www.w3.org/2001/XMLSchema-instance";
>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
>>> https://maven.apache.org/maven-v4_0_0.xsd";>
>>> <modelVersion>4.0.0</modelVersion>
>>> <groupId>spark</groupId>
>>> <version>3.0</version>
>>> <name>${project.artifactId}</name>
>>>
>>> <properties>
>>> <maven.compiler.source>1.7</maven.compiler.source>
>>> <maven.compiler.target>1.7</maven.compiler.target>
>>> <encoding>UTF-8</encoding>
>>> <scala.version>2.12.17</scala.version>
>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>> </properties>
>>> <dependencies>
>>> <dependency>
>>> <groupId>org.scala-lang</groupId>
>>> <artifactId>scala-library</artifactId>
>>> <version>2.13.6</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-core_2.13</artifactId>
>>> <version>3.4.0</version>
>>> <scope>provided</scope>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-sql_2.13</artifactId>
>>> <version>3.4.0</version>
>>> <scope>provided</scope>
>>>
>>> The pom file is attached. These are the errors I am getting
>>>
>>> [ERROR] error: error while loading package, class file
>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)'
>>> is broken
>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>> of package.class: Scala signature package has wrong version
>>> [ERROR] error: error while loading package, class file
>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)'
>>> is broken
>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>> of package.class: Scala signature package has wrong version
>>> [ERROR] error: error while loading package, class file
>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)'
>>> is broken
>>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature
>>> of package.class: Scala signature package has wrong version
>>> [ERROR] three errors found
>>> [ERROR] Failed to execute goal
>>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project
>>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with
>>> an error: 1(Exit value: 1) -> [Help 1]
>>>
>>> Any ideas will be appreciated.
>>>
>>> Mich Talebzadeh,
>>> Lead Solutions Architect/Engineering Lead
>>> Palantir Technologies Limited
>>> London
>>> United Kingdom
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
