[
https://issues.apache.org/jira/browse/AMBARI-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated AMBARI-17149:
---------------------------------------
Description:
Issue 1 :
HadoopTimelineMetricsSink has and ExecutorService thread which is not a daemon
thread. The executor is not shutdown, and thread is not interrupted while the
sink stops. This causes region server process to hang although the rest of the
threads have all exited. The new thread should be marked as a daemon thread,
and also call shutdown() or shutdownNow() on the executor in sink close.
Issue 2:
Seeing in the RS logs during every HBase metrics system restart
{code}
2016-06-24 22:51:11,440 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded
properties from hadoop-metrics2-hbase.properties
2016-06-24 22:51:11,443 WARN [HBase-Metrics2-1] impl.MetricsSystemImpl: Error
creating sink 'ganglia'
org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin:
org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
at
org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:203)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:529)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:501)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:480)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:189)
at
org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:78)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Does not contain a valid
host:port authority: :8656
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
at org.apache.hadoop.metrics2.util.Servers.parse(Servers.java:61)
at
org.apache.hadoop.metrics2.sink.ganglia.AbstractGangliaSink.init(AbstractGangliaSink.java:135)
at
org.apache.hadoop.metrics2.sink.ganglia.GangliaSink30.init(GangliaSink30.java:59)
at org.apache.hado
{code}
Due to an if-else construct, hadoop-metrics2-hbase.properties is getting
populated with Ganglia Sink properties when AMS is not installed.
The fix is to change such that a cluster can be deployed with either Ganglia or
AMS or both.
was:
HadoopTimelineMetricsSink has and ExecutorService thread which is not a daemon
thread. The executor is not shutdown, and thread is not interrupted while the
sink stops. This causes region server process to hang although the rest of the
threads have all exited.
The new thread should be marked as a daemon thread, and also call shutdown() or
shutdownNow() on the executor in sink close.
> Issues in AMS HadoopTimelineMetricsSink integration with HBase metric system.
> -----------------------------------------------------------------------------
>
> Key: AMBARI-17149
> URL: https://issues.apache.org/jira/browse/AMBARI-17149
> Project: Ambari
> Issue Type: Bug
> Components: ambari-metrics
> Affects Versions: 2.4.0
> Reporter: Aravindan Vijayan
> Assignee: Aravindan Vijayan
> Priority: Critical
> Fix For: 2.4.0
>
> Attachments: AMBARI-17149-2.patch, AMBARI-17149.patch
>
>
> Issue 1 :
> HadoopTimelineMetricsSink has and ExecutorService thread which is not a
> daemon thread. The executor is not shutdown, and thread is not interrupted
> while the sink stops. This causes region server process to hang although the
> rest of the threads have all exited. The new thread should be marked as a
> daemon thread, and also call shutdown() or shutdownNow() on the executor in
> sink close.
> Issue 2:
> Seeing in the RS logs during every HBase metrics system restart
> {code}
> 2016-06-24 22:51:11,440 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded
> properties from hadoop-metrics2-hbase.properties
> 2016-06-24 22:51:11,443 WARN [HBase-Metrics2-1] impl.MetricsSystemImpl:
> Error creating sink 'ganglia'
> org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating
> plugin: org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
> at
> org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:203)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:529)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:501)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:480)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:189)
> at
> org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:78)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Does not contain a valid
> host:port authority: :8656
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
> at org.apache.hadoop.metrics2.util.Servers.parse(Servers.java:61)
> at
> org.apache.hadoop.metrics2.sink.ganglia.AbstractGangliaSink.init(AbstractGangliaSink.java:135)
> at
> org.apache.hadoop.metrics2.sink.ganglia.GangliaSink30.init(GangliaSink30.java:59)
> at org.apache.hado
> {code}
> Due to an if-else construct, hadoop-metrics2-hbase.properties is getting
> populated with Ganglia Sink properties when AMS is not installed.
> The fix is to change such that a cluster can be deployed with either Ganglia
> or AMS or both.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
