Thank you, Martin! I got it working now using the same shading rules in my
project as in Spark.
From: Martin Grund
Date: Monday, 28. August 2023 at 17:58
To: Stefan Hagedorn
Cc: dev@spark.apache.org
Subject: Re: Spark Connect: API mismatch in SparkSesession#execute
Hi Stefan,
There are some current limitations around how protobuf is embedded in Spark
Connect. One of the challenges there is that for compatibility reasons we
currently shade protobuf that then shades the `prototobuf.GeneramtedMessage`
class. The way to work around this is to shade the protobuf library in your
code following the same rules as in Spark.
I have a fully working example here:
https://github.com/grundprinzip/spark-connect-appstate-example/tree/main
We definitely looking forward to improve the usability.
Hope this helps,
Martin
On Mon, Aug 28, 2023 at 4:19 PM Stefan Hagedorn
mailto:stefan.haged...@actian.com>> wrote:
Hi everyone,
Trying my luck here, after no success in the user mailing list :)
I’m trying to use the "extension" feature of the Spark Connect CommandPlugin
(Spark 3.4.1) [1].
I created a simple protobuf message `MyMessage` that I want to send from the
connect client-side to the connect server (where I registered my plugin).
The source API for SparkSession class in `spark-connect-client-jvm` provides a
method `execute` that accepts a `com.google.protobuf.Any` [2], so I packed the
MyMessage object in an Any:
val spark = SparkSession.builder().remote("sc://localhost").build()
val cmd = com.test.MyMessage.newBuilder().setBlubb("hello world").build()
val googleAny = com.google.protobuf.Any.pack(cmd)
spark.execute(googleAny)
This compiles, but during execution I receive a NoSuchMethodError:
java.lang.NoSuchMethodError: 'void
org.apache.spark.sql.SparkSession.execute(com.google.protobuf.Any)'
After looking around for a while after decompiling I found that
spark-connect-client-jvm_2.12-3.4.1.jar!SparkSession#execute accepts a
`org.sparkproject.connect.client.com.google.protobuf.Any` (instead of only the
com.google.protobuf.Any).
Am I missing something, how am I supposed to use this? Is there an additional
build step or should I use a specific plugin? I'm using the sbt-protoc [3]
plugin in my setup.
Packing my message object `cmd` into an
org.sparkproject.connect.client.com.google.protobuf.Any does not compile.
Thanks,
Stefan
[1] https://github.com/apache/spark/pull/39291
[2]
https://github.com/apache/spark/blob/64c26b7cb9b4c770a3e056404e05f6b6603746ee/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala#L444
[3] https://github.com/thesamet/sbt-protoc