Hi Nimrod, We are working on this as we speak.
There is already a PR out for the extensions use case: https://github.com/apache/spark/pull/49604 Kind regards, Herman On Mon, Feb 3, 2025 at 10:10 AM Nimrod Ofek <ofek.nim...@gmail.com> wrote: > Hi, > > In https://spark.apache.org/spark-connect/ - at the bottom it says: > > Check out the guide on migrating from Spark JVM to Spark Connect to learn > more about how to write code that works with Spark Connect. Also, check out > how to build Spark Connect custom extensions to learn how to use > specialized logic. > > I think there should be links to those guides - I couldn't find such > guides. I did find some quick start that shows how to write in python/ > Scala - but not a migration guide (maybe I just didn't find it) - but more > importantly - I couldn't find anything about "how to build Spark Connect > custom extensions to learn how to use specialized logic". > Is there such a guide? If so - what are the cases where one would need it? > > On a side note - there is a small documentation bug here > <https://spark.apache.org/docs/latest/spark-connect-overview.html#what-is-supported-in-spark-34> > - > it states Spark 3.4 although we are already 3.5 and soon 4.0. > > > Thanks! > Nimrod >
