[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356393#comment-16356393
]
Sahil Takiar commented on HIVE-18034:
-------------------------------------
Looks like the Spark Web UI uses the following logic to detect if excessive GC
time has been spent:
{code}
// totalDuration range from 0 to 50% GC time, alpha max = 1
function totalDurationAlpha(totalGCTime, totalDuration) {
return totalDuration > 0 ?
(Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1;
}
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
var GCTimePercent = 0.1;
function totalDurationStyle(totalGCTime, totalDuration) {
// Red if GC time over GCTimePercent of total time
return (totalGCTime > GCTimePercent * totalDuration) ?
("hsla(0, 100%, 50%, " + totalDurationAlpha(totalGCTime, totalDuration)
+ ")") : "";
{code}
We should be able to use the info in {{o.a.hive.spark.client.metrics}} to do
the same calculation. {{executorRunTime}} + {{ executorDeserializeTime}} should
give us the total executor duration, and {{jvmGCTime}} should give the GC time.
I think a {{LOG#warn}} and a warning on the console should be printed.
> Improving logging with HoS executors spend lots of time in GC
> -------------------------------------------------------------
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Priority: Major
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
