[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[ https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036644#comment-16036644 ] Lefty Leverenz commented on HIVE-16343: --- [~prasanth_j], so far the particular LLAP metrics haven't been documented. But should they be? And if so, where -- the LLAP design doc or the Metrics doc? * [LLAP -- Monitoring | https://cwiki.apache.org/confluence/display/Hive/LLAP#LLAP-Monitoring] * [Hive Metrics | https://cwiki.apache.org/confluence/display/Hive/Hive+Metrics] > LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring > > > Key: HIVE-16343 > URL: https://issues.apache.org/jira/browse/HIVE-16343 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-16343.1.patch, HIVE-16343.2.patch > > > Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful > for monitoring and also setting up triggers via JMC. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[ https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001923#comment-16001923 ] Siddharth Seth commented on HIVE-16343: --- +1. I'd still test the smap approach for perf. > LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring > > > Key: HIVE-16343 > URL: https://issues.apache.org/jira/browse/HIVE-16343 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16343.1.patch, HIVE-16343.2.patch > > > Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful > for monitoring and also setting up triggers via JMC. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[
https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999599#comment-15999599
]
Hive QA commented on HIVE-16343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866693/HIVE-16343.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=226)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5085/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5085/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5085/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12866693 - PreCommit-HIVE-Build
> LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
>
>
> Key: HIVE-16343
> URL: https://issues.apache.org/jira/browse/HIVE-16343
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16343.1.patch, HIVE-16343.2.patch
>
>
> Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful
> for monitoring and also setting up triggers via JMC.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[ https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999132#comment-15999132 ] Prasanth Jayachandran commented on HIVE-16343: -- Removed all other options to find PID. Using JVM_PID approach suggested by [~sseth]. Wrt. SMAP discussed with [~rajesh.balamohan], looks like it won't be in the order of multiple seconds. Also NM metrics dumps meminfo every few seconds (LLAP dump interval is even slower). > LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring > > > Key: HIVE-16343 > URL: https://issues.apache.org/jira/browse/HIVE-16343 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16343.1.patch, HIVE-16343.2.patch > > > Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful > for monitoring and also setting up triggers via JMC. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[ https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981707#comment-15981707 ] Siddharth Seth commented on HIVE-16343: --- This look up can be quite expensive. e.g. the SMAPS based lookup can take multiple seconds. I don't think refreshing it every 10s is a good idea. Need to have some kind of guard around when it gets refreshed (independent of the metrics config) > LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring > > > Key: HIVE-16343 > URL: https://issues.apache.org/jira/browse/HIVE-16343 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16343.1.patch > > > Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful > for monitoring and also setting up triggers via JMC. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[
https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981631#comment-15981631
]
Prasanth Jayachandran commented on HIVE-16343:
--
bq. While launching the process. environment.put("JVM_PID", "$$") / export.
Within the process - System.getenv().get("JVM_PID").
does it also happen for process launched by slider?
bq. Is there an easier and more reliable way to do this, instead of relying on
a pid file
I thought this is the reliable when compared to others :) The current default
location for the pid file is not reliable as it defaults to /tmp/user location.
If JVM_PID is guaranteed to be set I can add that option as well.
bq. May want to introduce a config for which process monitor to use, instead of
relying on a YARN configuration.
hmm.. why do we need this? Unless LLAP adds its own class it is not that
useful. Isn't it?
bq. How often will the metrics be collected?
Configurable in hadoop-metrics2.properties file. The template and Ambari
default is to collect every 10s and publish every 5 mins.
> LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
>
>
> Key: HIVE-16343
> URL: https://issues.apache.org/jira/browse/HIVE-16343
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16343.1.patch
>
>
> Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful
> for monitoring and also setting up triggers via JMC.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[
https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981608#comment-15981608
]
Siddharth Seth commented on HIVE-16343:
---
Getting access to the PID. Is there an easier and more reliable way to do this,
instead of relying on a pid file. Tez/YARN use the following - While launching
the process. environment.put("JVM_PID", "$$") / export. Within the process -
System.getenv().get("JVM_PID").
If retaining the current method of accessing the pid file, please move to a
helper class. The daemon class is getting a little noisy.
May want to introduce a config for which process monitor to use, instead of
relying on a YARN configuration.
How often will the metrics be collected?
> LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
>
>
> Key: HIVE-16343
> URL: https://issues.apache.org/jira/browse/HIVE-16343
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16343.1.patch
>
>
> Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful
> for monitoring and also setting up triggers via JMC.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[
https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972385#comment-15972385
]
Hive QA commented on HIVE-16343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863769/HIVE-16343.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10580 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ptfgroupbyjoin]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4724/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4724/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4724/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863769 - PreCommit-HIVE-Build
> LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
>
>
> Key: HIVE-16343
> URL: https://issues.apache.org/jira/browse/HIVE-16343
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16343.1.patch
>
>
> Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful
> for monitoring and also setting up triggers via JMC.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16343) LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
[
https://issues.apache.org/jira/browse/HIVE-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972302#comment-15972302
]
Prasanth Jayachandran commented on HIVE-16343:
--
[~sseth]/[~gopalv] can someone plz take a look at this patch?
Verified with some more logging that PIDs are read correctly by all 3 methods
{code}
2017-04-18T03:48:07,736 INFO [main ()]
org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Read PID 26891 from
/grid/5/hadoop/yarn/local/usercache/prasanth/appcache/application_1490656001509_0727/container_e13_1490656001509_0727_01_09/app/run/llap-daemon.pid
2017-04-18T03:48:07,743 INFO [main ()]
org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: PID from readlink: 26891
2017-04-18T03:48:07,743 INFO [main ()]
org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: PID from jvm name: 26891
{code}
> LLAP: Publish YARN's ProcFs based memory usage to metrics for monitoring
>
>
> Key: HIVE-16343
> URL: https://issues.apache.org/jira/browse/HIVE-16343
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16343.1.patch
>
>
> Publish MemInfo from ProcfsBasedProcessTree to llap metrics. This will useful
> for monitoring and also setting up triggers via JMC.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
