You may want to make sure you include the jar of P4J and your plugins as
part of the following so that both the driver and executors have access.
If HDFS is out then you could
make a common mount point on each of the executor nodes so they have access
to the classes.
- spark-submit --jars /common/path/to/jars
- spark.driver.extraClassPath or it's alias --driver-class-path to set
extra classpaths on the node running the driver.
- spark.executor.extraClassPath to set extra class path on the Worker
nodes.
On Tue, Apr 21, 2020 at 1:13 AM Shashanka Balakuntala <
shbalakunt...@gmail.com> wrote:
> Hi users,
> I'm a bit of newbie to spark infrastructure. And i have a small doubt.
> I have a maven project with plugins generated separately in a folder and
> normal java command to run is as follows:
> `java -Dp4j.pluginsDir=./plugins -jar /path/to/jar`
>
> Now when I run this program in local with spark-submit with standalone
> cluster(not cluster mode) the program compiles and plugins are in "plugins"
> folder in the $SPARK_HOME and it is getting recognised.
> The same is not the case in cluster mode. It says the Extenstion point is
> not loaded. Please advise on how can i create a folder which can be shared
> among the workers in "plugin" folder.
>
> PS: HDFS is not an options as we dont have a different setup
>
> Thanks.
>
>
> *Regards*
> Shashanka Balakuntala Srinivasa
>
>
