[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304829#comment-15304829 ] Szehon Ho commented on HIVE-13376: -- Sounds like a good plan to me, thanks for digging into this. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303395#comment-15303395 ] Xuefu Zhang commented on HIVE-13376: Sounds good to me, [~lirui]. Disabling spark.yarn.submit.waitAppCompletion sounds good. However, I'm not sure if it has any other use other than checking app aliveness. Please find out. Thanks. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[
https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303379#comment-15303379
]
Rui Li commented on HIVE-13376:
---
[~xuefuz], [~szehon] - I just did more tests about this and want to correct
some of my previous comments:
# In yarn-cluster mode, {{SparkSubmit}} runs the {{Client}}. The Client keeps
checking the app state and printing the logs. On hive side, we read from
SparkSubmit's input and err streams and print to hive log.
# In yarn-client mode, {{SparkSubmit}} runs our {{RemoteDriver}}. RemoteDirver
waits for the app to start running and then serves the job requests from hive.
It doesn't report the app state after that.
# The verbose logging only happens with yarn-cluster mode.
# The long interval only affects yarn-client mode.
# To avoid the state reports in yarn-cluster mode, we can change log level
(e.g. WARN instead of INFO), or we can set
{{spark.yarn.submit.waitAppCompletion=false}} and {{SparkSubmit}} will
terminate after it submits the app to RM.
I'd prefer disabling {{spark.yarn.submit.waitAppCompletion}}, if it doesn't
cause any other trouble.
> HoS emits too many logs with application state
> --
>
> Key: HIVE-13376
> URL: https://issues.apache.org/jira/browse/HIVE-13376
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Fix For: 2.1.0
>
> Attachments: HIVE-13376.2.patch, HIVE-13376.patch
>
>
> The logs get flooded with something like:
> > Mar 28, 3:12:21.851 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:21.912 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:22.853 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:22.913 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:23.855 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> While this is good information, it is a bit much.
> Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be
> higher and perhaps made configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302634#comment-15302634 ] Xuefu Zhang commented on HIVE-13376: [~lirui], There are some suggestions in HIVE-12569, including your idea about log4j. Does any of them work? If it's hard to change on our end, can we fix on Spark side? Thanks. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302623#comment-15302623 ] Szehon Ho commented on HIVE-13376: -- Sorry I thought this interval only controlled logging and not the application state check. It would be much better if spark offered a separate knob for log interval, but I guess they do not unfortunately. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[
https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301613#comment-15301613
]
Rui Li commented on HIVE-13376:
---
Hi [~xuefuz], I just briefly looked at the code. Although there're switches to
control whether to log the app state, the switches are not exposed to user via
configurations. So I think in order to disable the logging, we either need a
log level higher than INFO, or we can disable
{{spark.yarn.submit.waitAppCompletion}} (only works for yarn-cluster) .
Otherwise we need the interval to avoid the verbose state logs. Let me know if
there's other method to achieve it.
Related code in {{Client.scala}}:
{code}
def monitorApplication(
appId: ApplicationId,
returnOnRunning: Boolean = false,
logApplicationReport: Boolean = true): (YarnApplicationState,
FinalApplicationStatus) = {
val interval = sparkConf.getLong("spark.yarn.report.interval", 1000)
var lastState: YarnApplicationState = null
while (true) {
Thread.sleep(interval)
val report: ApplicationReport =
try {
getApplicationReport(appId)
} catch {
case e: ApplicationNotFoundException =>
logError(s"Application $appId not found.")
return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
case NonFatal(e) =>
logError(s"Failed to contact YARN for application $appId.", e)
return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
}
val state = report.getYarnApplicationState
if (logApplicationReport) {
logInfo(s"Application report for $appId (state: $state)")
// If DEBUG is enabled, log report details every iteration
// Otherwise, log them every time the application changes state
if (log.isDebugEnabled) {
logDebug(formatReportDetails(report))
} else if (lastState != state) {
logInfo(formatReportDetails(report))
}
}
if (lastState != state) {
state match {
case YarnApplicationState.RUNNING =>
reportLauncherState(SparkAppHandle.State.RUNNING)
case YarnApplicationState.FINISHED =>
reportLauncherState(SparkAppHandle.State.FINISHED)
case YarnApplicationState.FAILED =>
reportLauncherState(SparkAppHandle.State.FAILED)
case YarnApplicationState.KILLED =>
reportLauncherState(SparkAppHandle.State.KILLED)
case _ =>
}
}
if (state == YarnApplicationState.FINISHED ||
state == YarnApplicationState.FAILED ||
state == YarnApplicationState.KILLED) {
cleanupStagingDir(appId)
return (state, report.getFinalApplicationStatus)
}
if (returnOnRunning && state == YarnApplicationState.RUNNING) {
return (state, report.getFinalApplicationStatus)
}
lastState = state
}
// Never reached, but keeps compiler happy
throw new SparkException("While loop is depleted! This should never
happen...")
}
{code}
> HoS emits too many logs with application state
> --
>
> Key: HIVE-13376
> URL: https://issues.apache.org/jira/browse/HIVE-13376
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Fix For: 2.1.0
>
> Attachments: HIVE-13376.2.patch, HIVE-13376.patch
>
>
> The logs get flooded with something like:
> > Mar 28, 3:12:21.851 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:21.912 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:22.853 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:22.913 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:23.855 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> While this is good information, it is a bit much.
> Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be
> higher and perhaps made configurable.
--
This message was sent by Atlassian
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300301#comment-15300301 ] Xuefu Zhang commented on HIVE-13376: Yeah. I meant that maybe we don't need to change the interval. Rather, we can change logging instead. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300260#comment-15300260 ] Rui Li commented on HIVE-13376: --- Spark checks the app state and then (optionally) logs the state report. No job is accepted before the app reaches RUNNING state. So if Spark waits for 60s before it checks the state, the first job will have a considerable start-up overhead. You can do some local tests to verify this. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300039#comment-15300039 ] Xuefu Zhang commented on HIVE-13376: I think the issue is about logging, not about state checking. I thought there were suggestions of disabling the log entry by putting things in log4j configurations. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299669#comment-15299669 ] Rui Li commented on HIVE-13376: --- I think the default 60s interval is a little too long. Spark will at least wait for that interval before it checks the app state, which is likely to increase the start-up overhead. What do you guys think? > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[
https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221135#comment-15221135
]
Hive QA commented on HIVE-13376:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12796198/HIVE-13376.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9906 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-groupby10.q-timestamp_comparison.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join0.q-union_remove_9.q-smb_mapjoin_21.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-date_udf.q-auto_join4.q-and-12-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7428/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7428/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7428/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12796198 - PreCommit-HIVE-TRUNK-Build
> HoS emits too many logs with application state
> --
>
> Key: HIVE-13376
> URL: https://issues.apache.org/jira/browse/HIVE-13376
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-13376.2.patch, HIVE-13376.patch
>
>
> The logs get flooded with something like:
> > Mar 28, 3:12:21.851 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:21.912 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:22.853 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:22.913 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:23.855 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> While this is good information, it is a bit much.
> Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be
> higher and perhaps made configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219230#comment-15219230 ] Xuefu Zhang commented on HIVE-13376: +1 on my side too. As a side note, this is a dupe of HIVE-12569, which I will mark accordingly. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219163#comment-15219163 ] Rui Li commented on HIVE-13376: --- Thanks [~szehon] for the update. +1. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13376.2.patch, HIVE-13376.patch > > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[
https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217289#comment-15217289
]
Rui Li commented on HIVE-13376:
---
Thanks [~szehon] for the fix!
I found the config in spark code but not in its official doc. Have you verified
the effects of the patch, for both yarn-client and yarn-cluster mode?
Besides, would you mind add the config in {{HiveSparkClientFactory}} instead of
{{SparkClientImpl}}? I think most spark configs are set there.
> HoS emits too many logs with application state
> --
>
> Key: HIVE-13376
> URL: https://issues.apache.org/jira/browse/HIVE-13376
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-13376.patch
>
>
> The logs get flooded with something like:
> > Mar 28, 3:12:21.851 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:21.912 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:22.853 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> > Mar 28, 3:12:22.913 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report
> > for application_1458679386200_0149 (state: RUNNING)
> > Mar 28, 3:12:23.855 PMINFO
> > org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report
> > for application_1458679386200_0161 (state: RUNNING)
> While this is good information, it is a bit much.
> Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be
> higher and perhaps made configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217131#comment-15217131 ] Szehon Ho commented on HIVE-13376: -- Actually I made a mistake, this is controlled by the spark-submit. It is 'spark.yarn.report.interval' config that we need to play with. > HoS emits too many logs with application state > -- > > Key: HIVE-13376 > URL: https://issues.apache.org/jira/browse/HIVE-13376 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Szehon Ho >Assignee: Szehon Ho > > The logs get flooded with something like: > > Mar 28, 3:12:21.851 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:21.912 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:22.853 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > > Mar 28, 3:12:22.913 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report > > for application_1458679386200_0149 (state: RUNNING) > > Mar 28, 3:12:23.855 PMINFO > > org.apache.hive.spark.client.SparkClientImpl > > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report > > for application_1458679386200_0161 (state: RUNNING) > While this is good information, it is a bit much. > Seems like SparkJobMonitor hard-codes its interval to 1 second. It should be > higher and perhaps made configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
