[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[
https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830804#comment-17830804
]
slankka commented on SPARK-21711:
-
Thanks to [~mahesh_ambule] and Good to know, it's annoying to see errors in
launching outputs during submisstion on client machine.
{code:java}
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /stdout (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:133)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at
org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207){code}
while settings in log4j.properties like:
{code:java}
appender.file_appender.fileName=${spark.yarn.app.container.log.dir}/stdout{code}
h3. Conclusion
1. SPARK_SUBMIT_OPTS solves the problem above: client log should output to
correct directory.
2. setting SPARK_SUBMIT_OPTS of cause will NOT affect driver options or
executor options.
h3. Notes
modifing bin/spark-class like below won't work.
{code:java}
"$RUNNER" -Dlog4j.properties= -cp "$LAUNCH_CLASSPATH"
org.apache.spark.launcher.Main "$@"_ {code}
because the real submit command is partially built by
{code:java}
org.apache.spark.launcher.AbstractCommandBuilder#buildJavaCommand {code}
*buildJavaCommand* generates command starts from java executable to classpath
before
*org.apache.spark.deploy.SparkSubmit*
Logging and debuging
[Running Spark on YARN - Spark 3.5.1 Documentation
(apache.org)|https://spark.apache.org/docs/latest/running-on-yarn.html#debugging-your-application]
> spark-submit command should accept log4j configuration parameters for spark
> client logging.
> ---
>
> Key: SPARK-21711
> URL: https://issues.apache.org/jira/browse/SPARK-21711
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
>Affects Versions: 1.6.0, 2.1.0
>Reporter: Mahesh Ambule
>Priority: Minor
> Attachments: spark-submit client logs.txt
>
>
> Currently, log4j properties can be specified in spark 'conf' directory in
> log4j.properties file.
> The spark-submit command can override these log4j properties for driver and
> executors.
> But it can not override these log4j properties for *spark client *
> application.
> The user should be able to pass log4j properties for spark client using the
> spark-submit command.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124529#comment-16124529 ] Mahesh Ambule commented on SPARK-21711: --- Sean Owen: I figured out a way to configure the log4j configuration for spark-client/launcher. The $SPARK_SUBMIT_OPTS environment variable can be set to include log4j configuration. $SPARK_SUBMIT_OPTS gets appended to command line options when "org.apache.spark.deploy.SparkSubmit" is invoked. export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dlog4j.configuration=/home/mahesh/log4j.config" Link for the relevant code: https://github.com/apache/spark/blob/branch-2.1/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L242 > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124457#comment-16124457 ] Mahesh Ambule commented on SPARK-21711: --- My Application > spark-submit > spark client ---> yarn client ---> driver ---> executors I want to configure log4j Java options for spark client and yarn client, but spark-submit does not have the option to configure it. > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123967#comment-16123967 ] Mahesh Ambule commented on SPARK-21711: --- Here by spark client, I meant java client process started by spark-submit. I want to provide Java options to that process. I am talking about java client which invokes yarn client process and launches the driver and executor processes. > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123957#comment-16123957 ] Sean Owen commented on SPARK-21711: --- Oh, well that's your own application. You configure your own logging however you want. I didn't think that's what you're asking, because it's not a Spark issue. > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123941#comment-16123941 ] Mahesh Ambule commented on SPARK-21711: --- I dont want to configure executor or driver java options. I want to configure spark client java options. > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123911#comment-16123911 ] Sean Owen commented on SPARK-21711: --- It does, I'm referring to the spark.executor.extraJavaOptions config parameter. > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123889#comment-16123889 ] Mahesh Ambule commented on SPARK-21711: --- sean owen: Thanks for the reply. I tried to pass '-Dlog4j.configuration=' param to spark-submit command but it did not configure spark client logging. Log4j configuration parameters passed through spark-submit command are getting configured for driver and executor JVMs but not for the spark client JVM. Here I am more interested in spark client application logs. For more clarity, please find attached the sample spark client logs file. I went through spark-submit and related script files and found that spark-submit does not provide the option to pass parameters to client application JVM. Below is the code snippet from '$spark_home/bin/spark-class' file. This 'spark-class' file gets invoked by spark-submit. _ "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"_ Here $RUNNER is 'java' command and '$@' are arguments passed to spark-submit. There is no way to pass parameters to client java JVM. Is my understanding correct? And can the option be provided to pass parameters to client Java JVM? > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > Attachments: spark-submit client logs.txt > > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21711) spark-submit command should accept log4j configuration parameters for spark client logging.
[ https://issues.apache.org/jira/browse/SPARK-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123770#comment-16123770 ] Sean Owen commented on SPARK-21711: --- How about configuring the log4j config with -Dlog4j.configuration=... as a command line arg to the JVM? or packaging it in your app? I thought it picked up what you package in your application. > spark-submit command should accept log4j configuration parameters for spark > client logging. > --- > > Key: SPARK-21711 > URL: https://issues.apache.org/jira/browse/SPARK-21711 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.0, 2.1.0 >Reporter: Mahesh Ambule >Priority: Minor > > Currently, log4j properties can be specified in spark 'conf' directory in > log4j.properties file. > The spark-submit command can override these log4j properties for driver and > executors. > But it can not override these log4j properties for *spark client * > application. > The user should be able to pass log4j properties for spark client using the > spark-submit command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
