To me it seems like it's the best possible use case for PF4J.
[image: facebook] <https://fb.com/asm0dey> [image: twitter] <https://twitter.com/asm0di0> [image: linkedin] <https://linkedin.com/in/asm0dey> [image: instagram] <https://instagram.com/asm0dey> Pasha Finkelshteyn Developer Advocate for Data Engineering JetBrains asm0...@jetbrains.com https://linktr.ee/asm0dey Find out more <https://jetbrains.com> On Tue, 28 Nov 2023 at 12:47, Holden Karau <holden.ka...@gmail.com> wrote: > So I don’t think we make any particular guarantees around class path > isolation there, so even if it does work it’s something you’d need to pay > attention to on upgrades. Class path isolation is tricky to get right. > > On Mon, Nov 27, 2023 at 2:58 PM Faiz Halde <haldef...@gmail.com> wrote: > >> Hello, >> >> We are using spark 3.5.0 and were wondering if the following is >> achievable using spark-core >> >> Our use case involves spinning up a spark cluster where the driver >> application loads user jars containing spark transformations at runtime. A >> single spark application can load multiple user jars ( same cluster ) that >> can have class path conflicts if care is not taken >> >> AFAIK, to get this right requires the Executor to be designed in a way >> that allows for class path isolation ( UDF, lambda expressions ). Ideally >> per Spark Session is what we want >> >> I know Spark connect has been designed this way but Spark connect is not >> an option for us at the moment. I had some luck using a private method >> inside spark called JobArtifactSet.withActiveJobArtifactState >> >> Is it sufficient for me to run the user code enclosed >> within JobArtifactSet.withActiveJobArtifactState to achieve my requirement? >> >> Thank you >> >> >> Faiz >> >
