[GitHub] storm issue #2554: STORM-2939 add WorkerMetricsProcessor interface
Github user agresch commented on the issue: https://github.com/apache/storm/pull/2554 That is correct. With the current default RocksDB implementation, we would have workers forward to the Supervisor, and the Supervisor metric processor would be NimbusMetricProcessor - sending them to Nimbus for insertion into RocksDB. ---
[GitHub] storm issue #2554: STORM-2939 add WorkerMetricsProcessor interface
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2554 @agresch > We're planning on getting the metrics forwarded from the workers to the Supervisor and inserting from HBase there (or forwarding to Nimbus for RocksDB). In the non-HBase implementation, we need to be able to have a path from the workers to Nimbus. Once you intend WorkerMetricsProcessor to be run in Supervisor, non-HBase implementation should also forward metrics to Supervisor to delegate forwarding, hence the path would be worker -> Supervisor -> Nimbus, right? Just be clear about this. ---
[GitHub] storm issue #2554: STORM-2939 add WorkerMetricsProcessor interface
Github user agresch commented on the issue: https://github.com/apache/storm/pull/2554 1) My intent was that the metricstore would be for insertion, and the metricprocessor for either forwarding the metrics to the metricstore or just extending a metricstore and doing the metric insertion in place. The HBase metricprocessor implementation I'm working on does the latter, inserting directly from the Supervisor. 2) I do not have a plan for this yet, but we plan on tackling this. We're planning on getting the metrics forwarded from the workers to the Supervisor and inserting from HBase there (or forwarding to Nimbus for RocksDB). In the non-HBase implementation, we need to be able to have a path from the workers to Nimbus. ---
[GitHub] storm issue #2554: STORM-2939 add WorkerMetricsProcessor interface
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2554 Some questions: 1. Will we support combination of metricstore and metricprocessor? For example, NimbusMetricProcessor as processor but using HBaseStore as store. If metric store and metric processor should be selected as a pair to make it work, we may want to have less redundant way to configure. 2. I think we will eventually migrate worker built-in metrics. Do you have a plan or (sketched) idea of that? Because if we insert metrics to HBase store directly, it will require HBase dependencies in classpath. While it might be fine for Supervisor, sometimes it might not fine for user topologies. In this case, requiring workers to push their metrics into HBase directly would make some kinds of pains, and they would want to push metrics from user topology to daemons (either Nimbus or Supervisor) to let them handle instead. ---
