zachtrong commented on issue #1055:
URL: https://github.com/apache/sedona/issues/1055#issuecomment-1857970628
I'm encountering this error on spark scala project, it's so weird that
gradle dependency shows that commons-text version is 1.10.0.
Here are the full logs.
```text
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.commons.text.StringSubstitutor.setEnableUndefinedVariableException(Z)Lorg/apache/commons/text/StringSubstitutor;
at
org.apache.spark.ErrorClassesJsonReader.getErrorMessage(ErrorClassesJSONReader.scala:49)
at
org.apache.spark.SparkThrowableHelper$.getMessage(SparkThrowableHelper.scala:55)
at
org.apache.spark.SparkThrowableHelper$.getMessage(SparkThrowableHelper.scala:42)
at
org.apache.spark.sql.AnalysisException.<init>(AnalysisException.scala:80)
at
org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:52)
at org.apache.spark.sql.Dataset.write(Dataset.scala:3833)
```
```gradle
plugins {
id 'java'
}
ext {
scalaVersion = "2.12"
sparkVersion = "3.4.1"
}
group 'com.vdtdata'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'scala'
repositories {
jcenter()
mavenLocal()
mavenCentral()
maven {
url "https://oss.sonatype.org/content/repositories/snapshots";
}
maven {
url "https://repository.cloudera.com/artifactory/cloudera-repos/";
}
}
dependencies {
implementation
"org.scala-lang:scala-library:${project.ext.scalaVersion}.10"
implementation
"org.scala-lang:scala-reflect:${project.ext.scalaVersion}.10"
implementation
"org.scala-lang:scala-compiler:${project.ext.scalaVersion}.10"
implementation
"org.apache.hudi:hudi-spark3.4-bundle_${project.ext.scalaVersion}:0.14.0"
implementation "org.apache.hadoop:hadoop-aws:3.3.4"
implementation("org.apache.sedona:sedona-spark-shaded-3.4_${project.ext.scalaVersion}:1.5.0")
implementation("org.datasyslab:geotools-wrapper:1.5.0-28.2")
implementation("org.postgresql:postgresql:42.7.1")
implementation
"org.apache.spark:spark-mllib_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation
"org.apache.spark:spark-sql_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation
"org.apache.spark:spark-graphx_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation
"org.apache.spark:spark-launcher_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation
"org.apache.spark:spark-catalyst_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation
"org.apache.spark:spark-streaming_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation
"org.apache.spark:spark-core_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation
"org.apache.spark:spark-hive_${project.ext.scalaVersion}:${project.ext.sparkVersion}"
implementation 'org.apache.commons:commons-text:1.10.0' // also try
removing this line without avail
implementation group: 'commons-io', name: 'commons-io', version: '2.11'
}
```
```bash
gradle -q dependencyInsight --dependency org.apache.commons:commons-text
org.apache.commons:commons-text:1.10.0
Variant compile:
| Attribute Name | Provided | Requested |
|--------------------------------|----------|--------------|
| org.gradle.status | release | |
| org.gradle.category | library | library |
| org.gradle.libraryelements | jar | classes |
| org.gradle.usage | java-api | java-api |
| org.gradle.dependency.bundling | | external |
| org.gradle.jvm.environment | | standard-jvm |
| org.gradle.jvm.version | | 8 |
org.apache.commons:commons-text:1.10.0
+--- compileClasspath
\--- org.apache.spark:spark-core_2.12:3.4.1
+--- compileClasspath
+--- org.apache.spark:spark-mllib_2.12:3.4.1
| \--- compileClasspath
+--- org.apache.spark:spark-streaming_2.12:3.4.1
| +--- compileClasspath
| \--- org.apache.spark:spark-mllib_2.12:3.4.1 (*)
+--- org.apache.spark:spark-hive_2.12:3.4.1
| \--- compileClasspath
+--- org.apache.spark:spark-sql_2.12:3.4.1
| +--- compileClasspath
| +--- org.apache.spark:spark-mllib_2.12:3.4.1 (*)
| \--- org.apache.spark:spark-hive_2.12:3.4.1 (*)
+--- org.apache.spark:spark-graphx_2.12:3.4.1
| +--- compileClasspath
| \--- org.apache.spark:spark-mllib_2.12:3.4.1 (*)
\--- org.apache.spark:spark-catalyst_2.12:3.4.1
+--- compileClasspath
\--- org.apache.spark:spark-sql_2.12:3.4.1 (*)
(*) - Indicates repeated occurrences of a transitive dependency subtree.
Gradle expands transitive dependency subtrees only once per project; repeat
occurrences only display the root of the subtree, followed by this annotation.
A web-based, searchable dependency report is available by adding the --scan
option.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
