Hello there, I just upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload and got the error (java.lang.AbstractMethodError) never seen before; check the error stack attached in (a) bellow.
anyone knows if spark 2.3.1 works well with kafka spark-streaming-kafka-0-10? this link spark kafka integration page doesn't say anything about any limitation: https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html but this discussion seems to say there is indeed an issue when upgrading to spark 2.3.1: https://stackoverflow.com/questions/49180931/abstractmethoderror-creating-kafka-stream i also rebuilt the workload with some spark 2.3.1 jars (see (b) below). it doesn't seem to help. Would be great if anyone could kindly share any insights here. Thanks! Peter (a) the exception Exception in thread "stream execution thread for [id = 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId = 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99) at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializeLogIfNecessary(KafkaSourceProvider.scala:369) at org.apache.spark.internal.Logging$class.log(Logging.scala:46) at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(KafkaSourceProvider.scala:369) at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58) at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(KafkaSourceProvider.scala:369) at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpdater.set(KafkaSourceProvider.scala:439) at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaParamsForDriver(KafkaSourceProvider.scala:394) at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:90) at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:277) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77) at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194) at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:77) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:75) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266) at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:75) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61) at org.apache.spark.sql.execution.streaming.StreamExecution.org $apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:265) at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189) (b)* the build script update:* [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff build.sbt spk211-build.sbt.original 10,11c10,11 < libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %* "2.3.1"* < libraryDependencies += "org.apache.spark" % "spark-core_2.11" %* "2.3.1"* --- > libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1" > libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1" [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
