[jira] [Comment Edited] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830804#comment-17830804 ] slankka edited comment on SPARK-21711 at 3/26/24 6:47 AM: -- Thanks to [~mahesh_ambule] and Good to know, it's annoying to see errors in launching outputs during submission on client machine. {code:java} log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /stdout (Permission denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream.(FileOutputStream.java:133) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207){code} while settings in log4j.properties like: {code:java} appender.file_appender.fileName=${spark.yarn.app.container.log.dir}/stdout{code} h3. Conclusion 1. SPARK_SUBMIT_OPTS solves the problem above: client log should output to correct directory. 2. setting SPARK_SUBMIT_OPTS of cause will NOT affect driver options or executor options. h3. Notes modifing bin/spark-class like below won't work. {code:java} "$RUNNER" -Dlog4j.properties= -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ {code} because the real submit command is partially built by {code:java} org.apache.spark.launcher.AbstractCommandBuilder#buildJavaCommand {code} *buildJavaCommand* generates command starts from java executable to classpath before *org.apache.spark.deploy.SparkSubmit* Logging and debuging [Running Spark on YARN - Spark 3.5.1 Documentation (apache.org)|https://spark.apache.org/docs/latest/running-on-yarn.html#debugging-your-application] was (Author: adrian z): Thanks to [~mahesh_ambule] and Good to know, it's annoying to see errors in launching outputs during submisstion on client machine. {code:java} log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /stdout (Permission denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream.(FileOutputStream.java:133) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207){code} while settings in log4j.properties like: {code:java} appender.file_appender.fileName=${spark.yarn.app.container.log.dir}/stdout{code} h3. Conclusion 1. SPARK_SUBMIT_OPTS solves the problem above: client log should output to correct directory. 2. setting SPARK_SUBMIT_OPTS of cause will NOT affect driver options or executor options. h3. Notes modifing bin/spark-class like below won't work. {code:java} "$RUNNER" -Dlog4j.properties= -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ {code} because the real submit command is partially built by {code:java} org.apache.spark.launcher.AbstractCommandBuilder#buildJavaCommand {code} *buildJavaCommand* generates command starts from java executable to classpath before *org.apache.spark.deploy.SparkSubmit* Logging and debuging [Running Spark on YARN - Spark 3.5.1 Documentation (apache.org)|https://spark.apache.org/docs/latest/running-on-yarn.html#debugging-your-application] > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123967#comment-16123967 ] Mahesh Ambule edited comment on SPARK-21711 at 8/11/17 8:18 PM: Here by spark client, I meant java client process started by spark-submit. I want to provide Java options to this client process. I am talking about java client which invokes yarn client process and launches the driver and executor processes. was (Author: mahesh_ambule): Here by spark client, I meant java client process started by spark-submit. I want to provide Java options to that process. I am talking about java client which invokes yarn client process and launches the driver and executor processes. > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123889#comment-16123889 ] Mahesh Ambule edited comment on SPARK-21711 at 8/11/17 7:36 PM: @Sean Owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file. I went through spark-submit and related script files and found that spark-submit does not provide the option to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file. This 'spark-class' file gets invoked by spark-submit. _ "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit. There is no way to pass parameters to this java JVM. Is my understanding correct? And can the option be provided to pass parameters to client Java JVM? was (Author: mahesh_ambule): Sean Owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file. I went through spark-submit and related script files and found that spark-submit does not provide the option to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file. This 'spark-class' file gets invoked by spark-submit. _ "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit. There is no way to pass parameters to this java JVM. Is my understanding correct? And can the option be provided to pass parameters to client Java JVM? > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123889#comment-16123889 ] Mahesh Ambule edited comment on SPARK-21711 at 8/11/17 7:27 PM: Sean Owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file. I went through spark-submit and related script files and found that spark-submit does not provide the option to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file. This 'spark-class' file gets invoked by spark-submit. _ "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit. There is no way to pass parameters to this java JVM. Is my understanding correct? And can the option be provided to pass parameters to client Java JVM? was (Author: mahesh_ambule): Sean Owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file. I went through spark-submit and related script files and found that spark-submit does not provide the option to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file. This 'spark-class' file gets invoked by spark-submit. _ "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit. There is no way to pass parameters to client java JVM. Is my understanding correct? And can the option be provided to pass parameters to client Java JVM? > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123889#comment-16123889 ] Mahesh Ambule edited comment on SPARK-21711 at 8/11/17 7:22 PM: Sean Owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file. I went through spark-submit and related script files and found that spark-submit does not provide the option to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file. This 'spark-class' file gets invoked by spark-submit. _ "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit. There is no way to pass parameters to client java JVM. Is my understanding correct? And can the option be provided to pass parameters to client Java JVM? was (Author: mahesh_ambule): sean owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file. I went through spark-submit and related script files and found that spark-submit does not provide the option to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file. This 'spark-class' file gets invoked by spark-submit. _ "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit. There is no way to pass parameters to client java JVM. Is my understanding correct? And can the option be provided to pass parameters to client Java JVM? > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org