Re: Ambari Metrics Collector failing...
Correct, it’s not critical. Metrics can help a bit with debugging things like node hotspots in Metron and in HDP generally, but it’s certainly not required to run. Simon > On 7 Sep 2017, at 16:13, zeo...@gmail.comwrote: > > I wouldn't consider it a show stopper myself, happy to be persuaded > otherwise. I'm not even 100% sure it's related to Metron. I just put it > in Ambari's maintenance mode for now. > > Jon > > On Thu, Sep 7, 2017, 11:11 Laurens Vets wrote: > >> Is this something we need to fix for 0.4.1? Also, should I create JIRA >> ticket? >> >> On 2017-09-06 16:45, zeo...@gmail.com wrote: >>> I'm seeing the same issue right now as well on my fresh bare metal >>> install >>> of HDP (no Metron yet), haven't dug into it further to troubleshoot. >>> >>> Jon >>> >>> On Wed, Sep 6, 2017, 18:22 Laurens Vets wrote: >>> In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari and stops. Any idea what might be going on or where I can start troubleshooting this? In /var/log/hbase-ams-master-metron1.log I see lots of: 2017-09-06 21:49:16,469 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2017-09-06 21:49:16,477 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, serviceName = ams-hbase 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://metron1:6188/ws/v1/timeline/metrics 2017-09-06 21:49:16,491 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2017-09-06 21:49:16,518 WARN [HBase-Metrics2-1] lib.Interns: Metrics intern cache overflow at 2011 for MetricsSystem={MetricsSystem=MetricsInfo Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem record}} 2017-09-06 21:49:17,564 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x15e5929270d0001, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at >> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2017-09-06 21:49:17,565 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.0.0.11: 45058 which had sessionid 0x15e5929270d0001q Additionally, I also see: 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, serviceName = ams-hbase 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://metron1:6188/ws/v1/timeline/metrics 2017-09-06 22:19:16,470 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2017-09-06 22:20:26,491 INFO [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://metron1:6188/ws/v1/timeline/metrics This exceptions will be ignored for next 100 times 2017-09-06 22:20:26,491 WARN [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://metron1:6188/ws/v1/timeline/metrics Which ok because metrics collector isn't working... Any ideas? >> > -- > > Jon
Re: Ambari Metrics Collector failing...
I wouldn't consider it a show stopper myself, happy to be persuaded otherwise. I'm not even 100% sure it's related to Metron. I just put it in Ambari's maintenance mode for now. Jon On Thu, Sep 7, 2017, 11:11 Laurens Vetswrote: > Is this something we need to fix for 0.4.1? Also, should I create JIRA > ticket? > > On 2017-09-06 16:45, zeo...@gmail.com wrote: > > I'm seeing the same issue right now as well on my fresh bare metal > > install > > of HDP (no Metron yet), haven't dug into it further to troubleshoot. > > > > Jon > > > > On Wed, Sep 6, 2017, 18:22 Laurens Vets wrote: > > > >> In preparation of 0.4.1-rc, I'm trying to install the current github > >> master and I'm running into an issue with Ambari-metrics-collector. > >> "Metrics Collector" seems to start, but immediately turns red again > >> Ambari and stops. > >> > >> Any idea what might be going on or where I can start troubleshooting > >> this? > >> > >> In /var/log/hbase-ams-master-metron1.log I see lots of: > >> > >> 2017-09-06 21:49:16,469 INFO [HBase-Metrics2-1] impl.MetricsConfig: > >> loaded properties from hadoop-metrics2-hbase.properties > >> 2017-09-06 21:49:16,477 INFO [HBase-Metrics2-1] > >> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics > >> sink. > >> 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] > >> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, > >> serviceName = ams-hbase > >> 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] > >> timeline.HadoopTimelineMetricsSink: Collector Uri: > >> http://metron1:6188/ws/v1/timeline/metrics > >> 2017-09-06 21:49:16,491 INFO [HBase-Metrics2-1] > >> impl.MetricsSinkAdapter: Sink timeline started > >> 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] > >> impl.MetricsSystemImpl: > >> Scheduled snapshot period at 10 second(s). > >> 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] > >> impl.MetricsSystemImpl: > >> HBase metrics system started > >> 2017-09-06 21:49:16,518 WARN [HBase-Metrics2-1] lib.Interns: Metrics > >> intern cache overflow at 2011 for > >> MetricsSystem={MetricsSystem=MetricsInfo > >> Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem > >> record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem > >> record}} > >> 2017-09-06 21:49:17,564 WARN > >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: > >> caught end of stream exception > >> EndOfStreamException: Unable to read additional data from client > >> sessionid 0x15e5929270d0001, likely client has closed socket > >> at > >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > >> at > >> > >> > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) > >> at java.lang.Thread.run(Thread.java:745) > >> 2017-09-06 21:49:17,565 INFO > >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: > >> Closed socket connection for client /10.0.0.11: > >> 45058 which had sessionid 0x15e5929270d0001q > >> > >> Additionally, I also see: > >> > >> 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] > >> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics > >> sink. > >> 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] > >> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, > >> serviceName = ams-hbase > >> 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] > >> timeline.HadoopTimelineMetricsSink: Collector Uri: > >> http://metron1:6188/ws/v1/timeline/metrics > >> 2017-09-06 22:19:16,470 INFO [HBase-Metrics2-1] > >> impl.MetricsSinkAdapter: Sink timeline started > >> 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] > >> impl.MetricsSystemImpl: > >> Scheduled snapshot period at 10 second(s). > >> 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] > >> impl.MetricsSystemImpl: > >> HBase metrics system started > >> 2017-09-06 22:20:26,491 INFO [timeline] > >> timeline.HadoopTimelineMetricsSink: Unable to connect to collector, > >> http://metron1:6188/ws/v1/timeline/metrics > >> This exceptions will be ignored for next 100 times > >> > >> 2017-09-06 22:20:26,491 WARN [timeline] > >> timeline.HadoopTimelineMetricsSink: Unable to send metrics to > >> collector > >> by address:http://metron1:6188/ws/v1/timeline/metrics > >> > >> Which ok because metrics collector isn't working... > >> > >> Any ideas? > >> > -- Jon
Re: Ambari Metrics Collector failing...
Is this something we need to fix for 0.4.1? Also, should I create JIRA ticket? On 2017-09-06 16:45, zeo...@gmail.com wrote: I'm seeing the same issue right now as well on my fresh bare metal install of HDP (no Metron yet), haven't dug into it further to troubleshoot. Jon On Wed, Sep 6, 2017, 18:22 Laurens Vetswrote: In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari and stops. Any idea what might be going on or where I can start troubleshooting this? In /var/log/hbase-ams-master-metron1.log I see lots of: 2017-09-06 21:49:16,469 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2017-09-06 21:49:16,477 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, serviceName = ams-hbase 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://metron1:6188/ws/v1/timeline/metrics 2017-09-06 21:49:16,491 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2017-09-06 21:49:16,518 WARN [HBase-Metrics2-1] lib.Interns: Metrics intern cache overflow at 2011 for MetricsSystem={MetricsSystem=MetricsInfo Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem record}} 2017-09-06 21:49:17,564 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x15e5929270d0001, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2017-09-06 21:49:17,565 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.0.0.11: 45058 which had sessionid 0x15e5929270d0001q Additionally, I also see: 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, serviceName = ams-hbase 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://metron1:6188/ws/v1/timeline/metrics 2017-09-06 22:19:16,470 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2017-09-06 22:20:26,491 INFO [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://metron1:6188/ws/v1/timeline/metrics This exceptions will be ignored for next 100 times 2017-09-06 22:20:26,491 WARN [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://metron1:6188/ws/v1/timeline/metrics Which ok because metrics collector isn't working... Any ideas?
Re: Ambari Metrics Collector failing...
https://issues.apache.org/jira/browse/ZOOKEEPER-1582? On September 6, 2017 at 19:46:03, zeo...@gmail.com (zeo...@gmail.com) wrote: org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
Re: Ambari Metrics Collector failing...
I'm seeing the same issue right now as well on my fresh bare metal install of HDP (no Metron yet), haven't dug into it further to troubleshoot. Jon On Wed, Sep 6, 2017, 18:22 Laurens Vetswrote: > In preparation of 0.4.1-rc, I'm trying to install the current github > master and I'm running into an issue with Ambari-metrics-collector. > "Metrics Collector" seems to start, but immediately turns red again > Ambari and stops. > > Any idea what might be going on or where I can start troubleshooting > this? > > In /var/log/hbase-ams-master-metron1.log I see lots of: > > 2017-09-06 21:49:16,469 INFO [HBase-Metrics2-1] impl.MetricsConfig: > loaded properties from hadoop-metrics2-hbase.properties > 2017-09-06 21:49:16,477 INFO [HBase-Metrics2-1] > timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. > 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] > timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, > serviceName = ams-hbase > 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] > timeline.HadoopTimelineMetricsSink: Collector Uri: > http://metron1:6188/ws/v1/timeline/metrics > 2017-09-06 21:49:16,491 INFO [HBase-Metrics2-1] > impl.MetricsSinkAdapter: Sink timeline started > 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: > Scheduled snapshot period at 10 second(s). > 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: > HBase metrics system started > 2017-09-06 21:49:16,518 WARN [HBase-Metrics2-1] lib.Interns: Metrics > intern cache overflow at 2011 for > MetricsSystem={MetricsSystem=MetricsInfo > Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem > record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem > record}} > 2017-09-06 21:49:17,564 WARN > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: > caught end of stream exception > EndOfStreamException: Unable to read additional data from client > sessionid 0x15e5929270d0001, likely client has closed socket > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > at > > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) > at java.lang.Thread.run(Thread.java:745) > 2017-09-06 21:49:17,565 INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: > Closed socket connection for client /10.0.0.11: > 45058 which had sessionid 0x15e5929270d0001q > > Additionally, I also see: > > 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] > timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. > 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] > timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, > serviceName = ams-hbase > 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] > timeline.HadoopTimelineMetricsSink: Collector Uri: > http://metron1:6188/ws/v1/timeline/metrics > 2017-09-06 22:19:16,470 INFO [HBase-Metrics2-1] > impl.MetricsSinkAdapter: Sink timeline started > 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: > Scheduled snapshot period at 10 second(s). > 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: > HBase metrics system started > 2017-09-06 22:20:26,491 INFO [timeline] > timeline.HadoopTimelineMetricsSink: Unable to connect to collector, > http://metron1:6188/ws/v1/timeline/metrics > This exceptions will be ignored for next 100 times > > 2017-09-06 22:20:26,491 WARN [timeline] > timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector > by address:http://metron1:6188/ws/v1/timeline/metrics > > Which ok because metrics collector isn't working... > > Any ideas? > -- Jon
Ambari Metrics Collector failing...
In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari and stops. Any idea what might be going on or where I can start troubleshooting this? In /var/log/hbase-ams-master-metron1.log I see lots of: 2017-09-06 21:49:16,469 INFO [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2017-09-06 21:49:16,477 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, serviceName = ams-hbase 2017-09-06 21:49:16,478 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://metron1:6188/ws/v1/timeline/metrics 2017-09-06 21:49:16,491 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-09-06 21:49:16,500 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2017-09-06 21:49:16,518 WARN [HBase-Metrics2-1] lib.Interns: Metrics intern cache overflow at 2011 for MetricsSystem={MetricsSystem=MetricsInfo Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem record}} 2017-09-06 21:49:17,564 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x15e5929270d0001, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2017-09-06 21:49:17,565 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.0.0.11: 45058 which had sessionid 0x15e5929270d0001q Additionally, I also see: 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Identified hostname = metron1, serviceName = ams-hbase 2017-09-06 22:19:16,468 INFO [HBase-Metrics2-1] timeline.HadoopTimelineMetricsSink: Collector Uri: http://metron1:6188/ws/v1/timeline/metrics 2017-09-06 22:19:16,470 INFO [HBase-Metrics2-1] impl.MetricsSinkAdapter: Sink timeline started 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-09-06 22:19:16,471 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system started 2017-09-06 22:20:26,491 INFO [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://metron1:6188/ws/v1/timeline/metrics This exceptions will be ignored for next 100 times 2017-09-06 22:20:26,491 WARN [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://metron1:6188/ws/v1/timeline/metrics Which ok because metrics collector isn't working... Any ideas?