Re: Configuring logging properties for executor
Hi, I would like to report on trying the first option proposed by Lan - putting the log4j.properties file under the root of my application jar. It doesn't look like it is working on in my case: submitting the application to spark from the application code (not with spark-submit). It seems that in this case the executor doesn't see the log4j.properties that is located int he application jar and will use the default properties file. I can conclude it from the fact that the log is not created, and that's what I see in the executor console: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties I've also tried to add the below config to the application code, which didn't have any apparent influence: sparkConf.set(spark.executor.extraJavaOptions, -Dlog4j.configuration=log4j.properties); On Mon, Apr 20, 2015 at 5:59 PM, Lan Jiang ljia...@gmail.com wrote: Rename your log4j_special.properties file as log4j.properties and place it under the root of your jar file, you should be fine. If you are using Maven to build your jar, please the log4j.properties in the src/main/resources folder. However, please note that if you have other dependency jar file in the classpath that contains another log4j.properties file this way, it might not work since the first log4j.properties file that is loaded will be used. You can also do spark-submit —file log4j_special.properties … ,which should transfer your log4j property file to the worker nodes automatically without you copying them manually. Lan On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev michael...@gmail.com wrote: Hi all, I need to configure spark executor log4j.properties on a standalone cluster. It looks like placing the relevant properties file in the spark configuration folder and setting the spark.executor.extraJavaOptions from my application code: sparkConf.set(spark.executor.extraJavaOptions, -Dlog4j.configuration=log4j_special.properties); does the work, and the executor logs are written in the required place and level. As far as I understand, it works, because the spark configuration folder is on the class path, and passing parameter without path works here. However, I would like to avoid deploying these properties to each worker spark configuration folder. I wonder, if I put the properties in my application jar, is there any way of telling executor to load them? Thanks, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Configuring logging properties for executor
Rename your log4j_special.properties file as log4j.properties and place it under the root of your jar file, you should be fine. If you are using Maven to build your jar, please the log4j.properties in the src/main/resources folder. However, please note that if you have other dependency jar file in the classpath that contains another log4j.properties file this way, it might not work since the first log4j.properties file that is loaded will be used. You can also do spark-submit —file log4j_special.properties … ,which should transfer your log4j property file to the worker nodes automatically without you copying them manually. Lan On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev michael...@gmail.com wrote: Hi all, I need to configure spark executor log4j.properties on a standalone cluster. It looks like placing the relevant properties file in the spark configuration folder and setting the spark.executor.extraJavaOptions from my application code: sparkConf.set(spark.executor.extraJavaOptions, -Dlog4j.configuration=log4j_special.properties); does the work, and the executor logs are written in the required place and level. As far as I understand, it works, because the spark configuration folder is on the class path, and passing parameter without path works here. However, I would like to avoid deploying these properties to each worker spark configuration folder. I wonder, if I put the properties in my application jar, is there any way of telling executor to load them? Thanks, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Configuring logging properties for executor
Hi Lan, Thanks for fast response. It could be a solution if it works. I have more than one log4 properties file, for different run modes like debug/production, for executor and for application core. I think I would like to keep them separate. Then, I suppose I should give all other properties files a special names and keep the executor configuration with the default name? Can I conclude that going this way I will not be able to run several applications on the same cluster in parallel? Regarding submit, I am not using it now, I submit from the code, but I think I should consider this option. Thanks. On Mon, Apr 20, 2015 at 5:59 PM, Lan Jiang ljia...@gmail.com wrote: Rename your log4j_special.properties file as log4j.properties and place it under the root of your jar file, you should be fine. If you are using Maven to build your jar, please the log4j.properties in the src/main/resources folder. However, please note that if you have other dependency jar file in the classpath that contains another log4j.properties file this way, it might not work since the first log4j.properties file that is loaded will be used. You can also do spark-submit —file log4j_special.properties … ,which should transfer your log4j property file to the worker nodes automatically without you copying them manually. Lan On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev michael...@gmail.com wrote: Hi all, I need to configure spark executor log4j.properties on a standalone cluster. It looks like placing the relevant properties file in the spark configuration folder and setting the spark.executor.extraJavaOptions from my application code: sparkConf.set(spark.executor.extraJavaOptions, -Dlog4j.configuration=log4j_special.properties); does the work, and the executor logs are written in the required place and level. As far as I understand, it works, because the spark configuration folder is on the class path, and passing parameter without path works here. However, I would like to avoid deploying these properties to each worker spark configuration folder. I wonder, if I put the properties in my application jar, is there any way of telling executor to load them? Thanks, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Configuring logging properties for executor
Each application gets its own executor processes, so there should be no problem running them in parallel. Lan On Apr 20, 2015, at 10:25 AM, Michael Ryabtsev michael...@gmail.com wrote: Hi Lan, Thanks for fast response. It could be a solution if it works. I have more than one log4 properties file, for different run modes like debug/production, for executor and for application core. I think I would like to keep them separate. Then, I suppose I should give all other properties files a special names and keep the executor configuration with the default name? Can I conclude that going this way I will not be able to run several applications on the same cluster in parallel? Regarding submit, I am not using it now, I submit from the code, but I think I should consider this option. Thanks. On Mon, Apr 20, 2015 at 5:59 PM, Lan Jiang ljia...@gmail.com mailto:ljia...@gmail.com wrote: Rename your log4j_special.properties file as log4j.properties and place it under the root of your jar file, you should be fine. If you are using Maven to build your jar, please the log4j.properties in the src/main/resources folder. However, please note that if you have other dependency jar file in the classpath that contains another log4j.properties file this way, it might not work since the first log4j.properties file that is loaded will be used. You can also do spark-submit —file log4j_special.properties … ,which should transfer your log4j property file to the worker nodes automatically without you copying them manually. Lan On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev michael...@gmail.com mailto:michael...@gmail.com wrote: Hi all, I need to configure spark executor log4j.properties on a standalone cluster. It looks like placing the relevant properties file in the spark configuration folder and setting the spark.executor.extraJavaOptions from my application code: sparkConf.set(spark.executor.extraJavaOptions, -Dlog4j.configuration=log4j_special.properties); does the work, and the executor logs are written in the required place and level. As far as I understand, it works, because the spark configuration folder is on the class path, and passing parameter without path works here. However, I would like to avoid deploying these properties to each worker spark configuration folder. I wonder, if I put the properties in my application jar, is there any way of telling executor to load them? Thanks, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org mailto:user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org mailto:user-h...@spark.apache.org
Re: Configuring logging properties for executor
Oh, you are right, thanks. On Mon, Apr 20, 2015 at 6:31 PM, Lan Jiang ljia...@gmail.com wrote: Each application gets its own executor processes, so there should be no problem running them in parallel. Lan On Apr 20, 2015, at 10:25 AM, Michael Ryabtsev michael...@gmail.com wrote: Hi Lan, Thanks for fast response. It could be a solution if it works. I have more than one log4 properties file, for different run modes like debug/production, for executor and for application core. I think I would like to keep them separate. Then, I suppose I should give all other properties files a special names and keep the executor configuration with the default name? Can I conclude that going this way I will not be able to run several applications on the same cluster in parallel? Regarding submit, I am not using it now, I submit from the code, but I think I should consider this option. Thanks. On Mon, Apr 20, 2015 at 5:59 PM, Lan Jiang ljia...@gmail.com wrote: Rename your log4j_special.properties file as log4j.properties and place it under the root of your jar file, you should be fine. If you are using Maven to build your jar, please the log4j.properties in the src/main/resources folder. However, please note that if you have other dependency jar file in the classpath that contains another log4j.properties file this way, it might not work since the first log4j.properties file that is loaded will be used. You can also do spark-submit —file log4j_special.properties … ,which should transfer your log4j property file to the worker nodes automatically without you copying them manually. Lan On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev michael...@gmail.com wrote: Hi all, I need to configure spark executor log4j.properties on a standalone cluster. It looks like placing the relevant properties file in the spark configuration folder and setting the spark.executor.extraJavaOptions from my application code: sparkConf.set(spark.executor.extraJavaOptions, -Dlog4j.configuration=log4j_special.properties); does the work, and the executor logs are written in the required place and level. As far as I understand, it works, because the spark configuration folder is on the class path, and passing parameter without path works here. However, I would like to avoid deploying these properties to each worker spark configuration folder. I wonder, if I put the properties in my application jar, is there any way of telling executor to load them? Thanks, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html Sent from the Apache Spark User List mailing list archive at Nabble.com . - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
