[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[
https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787331#comment-15787331
]
Sean Owen commented on SPARK-18933:
---
stdout and stderr exist for all processes. You can redirect them as you like.
> Different log output between Terminal screen and stderr file
>
>
> Key: SPARK-18933
> URL: https://issues.apache.org/jira/browse/SPARK-18933
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation, Web UI
>Affects Versions: 1.6.3
> Environment: Yarn mode and standalone mode
>Reporter: Sean Wong
> Original Estimate: 612h
> Remaining Estimate: 612h
>
> First of all, I use the default log4j.properties in the Spark conf/
> But I found that the log output(e.g., INFO) is different between Terminal
> screen and stderr File. Some INFO logs exist in both of them. Some INFO logs
> exist in either of them. Why this happens? Is it supposed that the output
> logs are same between the terminal screen and stderr file?
> Then I did a Test. I modified the source code in SparkContext.scala and add
> one line log code "logInfo("This is textFile")" in the textFile function.
> However, after running an application, I found the log "This is textFile"
> shown in the terminal screen. no such log in the stderr file. I am not sure
> if this is a bug. So, hope you can solve this question. Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[
https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786822#comment-15786822
]
Sean Wong commented on SPARK-18933:
---
But there is no stderr or stdout file available for driver logs. Only executors
have these two files.
> Different log output between Terminal screen and stderr file
>
>
> Key: SPARK-18933
> URL: https://issues.apache.org/jira/browse/SPARK-18933
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation, Web UI
>Affects Versions: 1.6.3
> Environment: Yarn mode and standalone mode
>Reporter: Sean Wong
> Original Estimate: 612h
> Remaining Estimate: 612h
>
> First of all, I use the default log4j.properties in the Spark conf/
> But I found that the log output(e.g., INFO) is different between Terminal
> screen and stderr File. Some INFO logs exist in both of them. Some INFO logs
> exist in either of them. Why this happens? Is it supposed that the output
> logs are same between the terminal screen and stderr file?
> Then I did a Test. I modified the source code in SparkContext.scala and add
> one line log code "logInfo("This is textFile")" in the textFile function.
> However, after running an application, I found the log "This is textFile"
> shown in the terminal screen. no such log in the stderr file. I am not sure
> if this is a bug. So, hope you can solve this question. Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[
https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769604#comment-15769604
]
Sean Owen commented on SPARK-18933:
---
I disagree that it's even something that could be improved. Logs are local to
machine, yes. You can redirect or reconfigure driver logs / stdout / stderr
already.
> Different log output between Terminal screen and stderr file
>
>
> Key: SPARK-18933
> URL: https://issues.apache.org/jira/browse/SPARK-18933
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation, Web UI
>Affects Versions: 1.6.3
> Environment: Yarn mode and standalone mode
>Reporter: Sean Wong
> Original Estimate: 612h
> Remaining Estimate: 612h
>
> First of all, I use the default log4j.properties in the Spark conf/
> But I found that the log output(e.g., INFO) is different between Terminal
> screen and stderr File. Some INFO logs exist in both of them. Some INFO logs
> exist in either of them. Why this happens? Is it supposed that the output
> logs are same between the terminal screen and stderr file?
> Then I did a Test. I modified the source code in SparkContext.scala and add
> one line log code "logInfo("This is textFile")" in the textFile function.
> However, after running an application, I found the log "This is textFile"
> shown in the terminal screen. no such log in the stderr file. I am not sure
> if this is a bug. So, hope you can solve this question. Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[
https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768775#comment-15768775
]
Sean Wong commented on SPARK-18933:
---
Finally, I have got the answer. It's not the bug but the log system can be
improved.
The terminal screen output the log from the driver and the stderr file records
the log from each executor. The logs in the process of DAG construction are
shown in the terminal screen since these Loginfo codes are executed in the
driver. The logs in the compute() functions are recorded in the stderr file
since these Loginfo codes are executed within each executors.
Currently, unlike executor, the dirver does not have the stderr and stdout
file, so that the logs are only shown in the terminal screen. I think the log
system should also record the driver's logs in the stderr and stdout. I will
work on this in the future.
> Different log output between Terminal screen and stderr file
>
>
> Key: SPARK-18933
> URL: https://issues.apache.org/jira/browse/SPARK-18933
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation, Web UI
>Affects Versions: 1.6.3
> Environment: Yarn mode and standalone mode
>Reporter: Sean Wong
> Original Estimate: 612h
> Remaining Estimate: 612h
>
> First of all, I use the default log4j.properties in the Spark conf/
> But I found that the log output(e.g., INFO) is different between Terminal
> screen and stderr File. Some INFO logs exist in both of them. Some INFO logs
> exist in either of them. Why this happens? Is it supposed that the output
> logs are same between the terminal screen and stderr file?
> Then I did a Test. I modified the source code in SparkContext.scala and add
> one line log code "logInfo("This is textFile")" in the textFile function.
> However, after running an application, I found the log "This is textFile"
> shown in the terminal screen. no such log in the stderr file. I am not sure
> if this is a bug. So, hope you can solve this question. Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18933) Different log output between Terminal screen and stderr file
[
https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768739#comment-15768739
]
Sean Wong commented on SPARK-18933:
---
I found that Loginfo codes in the transformation() or actions() are only shown
in the terminal screen. These log codes are executed during the DAG
construction and before the job submission. But the Loginfo codes in the
compute() function of each RDD are only shown in the stderr file. These log
codes are executed in the process of real job execution. Now I know which part
of Loginfo go to terminal screen and which goes to stderr file. But I still do
not know why these will happen since they all use the Loginfo code and extends
Logging during the class definition.
> Different log output between Terminal screen and stderr file
>
>
> Key: SPARK-18933
> URL: https://issues.apache.org/jira/browse/SPARK-18933
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation, Web UI
>Affects Versions: 1.6.3
> Environment: Yarn mode and standalone mode
>Reporter: Sean Wong
> Original Estimate: 612h
> Remaining Estimate: 612h
>
> First of all, I use the default log4j.properties in the Spark conf/
> But I found that the log output(e.g., INFO) is different between Terminal
> screen and stderr File. Some INFO logs exist in both of them. Some INFO logs
> exist in either of them. Why this happens? Is it supposed that the output
> logs are same between the terminal screen and stderr file?
> Then I did a Test. I modified the source code in SparkContext.scala and add
> one line log code "logInfo("This is textFile")" in the textFile function.
> However, after running an application, I found the log "This is textFile"
> shown in the terminal screen. no such log in the stderr file. I am not sure
> if this is a bug. So, hope you can solve this question. Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
