In preparation of 0.4.1-rc, I'm trying to install the current github
master and I'm running into an issue with Ambari-metrics-collector.
"Metrics Collector" seems to start, but immediately turns red again
Ambari and stops.
Any idea what might be going on or where I can start troubleshooting
this?
In /var/log/hbase-ams-master-metron1.log I see lots of:
2017-09-06 21:49:16,469 INFO [HBase-Metrics2-1] impl.MetricsConfig:
loaded properties from hadoop-metrics2-hbase.properties
2017-09-06 21:49:16,477 INFO [HBase-Metrics2-1]
timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1]
timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
serviceName = ams-hbase
2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1]
timeline.HadoopTimelineMetricsSink: Collector Uri:
http://metron1:6188/ws/v1/timeline/metrics
2017-09-06 21:49:16,491 INFO [HBase-Metrics2-1]
impl.MetricsSinkAdapter: Sink timeline started
2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s).
2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl:
HBase metrics system started
2017-09-06 21:49:16,518 WARN [HBase-Metrics2-1] lib.Interns: Metrics
intern cache overflow at 2011 for
MetricsSystem={MetricsSystem=MetricsInfo
Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem
record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem
record}}
2017-09-06 21:49:17,564 WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
caught end of stream exception
EndOfStreamException: Unable to read additional data from client
sessionid 0x15e5929270d0001, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2017-09-06 21:49:17,565 INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
Closed socket connection for client /10.0.0.11:
45058 which had sessionid 0x15e5929270d0001q
Additionally, I also see:
2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1]
timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1]
timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
serviceName = ams-hbase
2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1]
timeline.HadoopTimelineMetricsSink: Collector Uri:
http://metron1:6188/ws/v1/timeline/metrics
2017-09-06 22:19:16,470 INFO [HBase-Metrics2-1]
impl.MetricsSinkAdapter: Sink timeline started
2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s).
2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl:
HBase metrics system started
2017-09-06 22:20:26,491 INFO [timeline]
timeline.HadoopTimelineMetricsSink: Unable to connect to collector,
http://metron1:6188/ws/v1/timeline/metrics
This exceptions will be ignored for next 100 times
2017-09-06 22:20:26,491 WARN [timeline]
timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector
by address:http://metron1:6188/ws/v1/timeline/metrics
Which ok because metrics collector isn't working...
Any ideas?
