We are trying to integrate spark 2.2.0 with Ignite 2.4 on kubernetes, and getting the following exception.
2018-06-06 03:23:51 INFO GridDiscoveryManager:495 - Topology snapshot [ver=3,
servers=2, clients=1, CPUs=12, offheap=3.1GB, heap=3.0GB]
Exception in thread "main" java.lang.NoSuchMethodError:
scala.util.matching.Regex.unapplySeq(Ljava/lang/CharSequence;)Lscala/Option;
at
com.fasterxml.jackson.module.scala.JacksonModule$.version$lzycompute(JacksonModule.scala:30)
at
com.fasterxml.jackson.module.scala.JacksonModule$.version(JacksonModule.scala:26)
at
com.fasterxml.jackson.module.scala.JacksonModule$class.version(JacksonModule.scala:49)
at
com.fasterxml.jackson.module.scala.DefaultScalaModule.version(DefaultScalaModule.scala:19)
at
com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:710)
at
org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at
org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
at org.apache.spark.SparkContext.parallelize(SparkContext.scala:718)
at
org.apache.spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:134)
at
org.apache.spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:146)
at
org.apache.ignite.examples.spark.SharedRDDExample.main(SharedRDDExample.java:81)
Two spark executors connecting to ignite in standalone mode. We are using
Ignite's sharedRDD example
(https://github.com/apache/ignite/blob/2.4.0/examples/src/main/spark/org/apache/ignite/examples/spark/SharedRDDExample.java)
The example jar and ignite was built with scala version 2.11. The ignite xml is
attached.
-----Original Message-----
From: Ray [mailto:[email protected]]
Sent: Friday, June 01, 2018 12:00 PM
To: [email protected]
Subject: Re: Spark + Ignite standalone mode on Kubernetes cluster.
Currently, Ignite supports Spark up to version 2.2.0.
Please try with Spark 2.2.0 or wait until this ticket is resolved.
https://issues.apache.org/jira/browse/IGNITE-8534
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
example_k8s.xml
Description: example_k8s.xml
