pegasas commented on issue #15548:
URL:
https://github.com/apache/dolphinscheduler/issues/15548#issuecomment-1924216467
> **Current workaround** for me is to pass `--master ... --deploy-mode
cluster` in the extra options. Since _spark-submit_ will use the last values,
this will send task to local cluster. For example look at this log which has my
own `--master` option which overrides Dolphin `--master local`:
>
> ```
> [INFO] 2024-02-02 14:27:38.934 +0000 - Final Shell file is :
> #!/bin/bash
> BASEDIR=$(cd `dirname $0`; pwd)
> cd $BASEDIR
> export SPARK_HOME=/opt/spark-3.5.0-bin-hadoop3
> export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
> ${SPARK_HOME}/bin/spark-submit --master local
> --class com.example.monitor.ScanMonitor --conf spark.driver.cores=1 --conf
spark.driver.memory=512M
> --conf spark.executor.instances=2 --conf spark.executor.cores=2
> --conf spark.executor.memory=2G
> --master spark://devel:7077 --deploy-mode cluster
>
file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/monitor-0.1-jdk11.jar
producer
> ...
> 24/02/02 14:27:54 INFO ClientEndpoint: Driver successfully submitted as
driver-20240202142754-0003
> 2024-02-02 14:28:00.038 +0000 - ->
> 24/02/02 14:27:59 INFO ClientEndpoint: State of driver-20240202142754-0003
is RUNNING
> 24/02/02 14:27:59 INFO ClientEndpoint: Driver running on
172.16.254.204:35595 (worker-20240202141308-172.16.254.204-35595)
> 24/02/02 14:27:59 INFO ClientEndpoint: spark-submit not configured to wait
for completion, exiting spark-submit JVM.
> ```
Thanks @git-blame , indeed extra parameters work, master is a important
parameter among spark as mentioned. I will communicate with community to see if
it is by design in previous discussions.
If not, I will add paramater into spark task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
