[
https://issues.apache.org/jira/browse/HIVE-12569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037025#comment-15037025
]
Nemon Lou commented on HIVE-12569:
----------------------------------
I noticed this log, too.
And here is my finds:
The log messages come from a spark class
org.apache.spark.deploy.yarn.Client.scala
{code}
logInfo(s"Application report for $appId (state: $state)")
{code}
And printed out by a Redirector in Hive's SparkClientImpl.java.
{code}
while ((line = in.readLine()) != null) {
LOG.info(line);
}
{code}
The direct way to fix it is only logging message when state changing (Need to
change spark code).
And another way is to add a log4j.properties file to spark-submit's class
path,changing org.apache.spark.deploy.yarn.Client.scala's log level to WARN.
May be there are other solutions.
> Excessive console message from SparkClientImpl [Spark Branch]
> -------------------------------------------------------------
>
> Key: HIVE-12569
> URL: https://issues.apache.org/jira/browse/HIVE-12569
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 2.0.0
> Reporter: Xuefu Zhang
> Assignee: Xuefu Zhang
> Priority: Blocker
>
> {code}
> 15/12/02 11:00:46 INFO client.SparkClientImpl: 15/12/02 11:00:46 INFO Client:
> Application report for application_1442517343449_0038 (state: RUNNING)
> 15/12/02 11:00:47 INFO client.SparkClientImpl: 15/12/02 11:00:47 INFO Client:
> Application report for application_1442517343449_0038 (state: RUNNING)
> 15/12/02 11:00:48 INFO client.SparkClientImpl: 15/12/02 11:00:48 INFO Client:
> Application report for application_1442517343449_0038 (state: RUNNING)
> 15/12/02 11:00:49 INFO client.SparkClientImpl: 15/12/02 11:00:49 INFO Client:
> Application report for application_1442517343449_0038 (state: RUNNING)
> 15/12/02 11:00:50 INFO client.SparkClientImpl: 15/12/02 11:00:50 INFO Client:
> Application report for application_1442517343449_0038 (state: RUNNING)
> {code}
> I see this using Hive CLI after a spark job is launched and it goes
> non-stopping even if the job is finished.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
