[GitHub] [hadoop] adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the visibility with Ozone Insight tool

2019-08-16 Thread GitBox
adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the 
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314651145
 
 

 ##
 File path: hadoop-ozone/insight/pom.xml
 ##
 @@ -0,0 +1,137 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
+https://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+  
+org.apache.hadoop
+hadoop-ozone
+0.5.0-SNAPSHOT
+  
+  hadoop-ozone-insight
+  0.5.0-SNAPSHOT
+  Apache Hadoop Ozone Insight Tool
+  Apache Hadoop Ozone Insight Tool
+  jar
+
+  
+
+  org.apache.hadoop
+  hadoop-ozone-ozone-manager
+
+
+  org.apache.hadoop
+  hadoop-ozone-common
+
+
+
+  org.apache.hadoop
+  hadoop-hdds-server-scm
+
+
+  org.apache.hadoop
+  hadoop-ozone-client
+
+
+  org.apache.hadoop
+  hadoop-ozone-filesystem
+
+
+  org.apache.hadoop
+  hadoop-hdds-server-framework
+
+
+  org.apache.hadoop
+  hadoop-common
+  compile
+
+
+  org.apache.hadoop
+  hadoop-hdfs
+  compile
+
+
+  com.sun.xml.bind
+  jaxb-core
+
+
+  javax.xml.bind
+  jaxb-api
+
+
+  javax.activation
+  activation
+
+
+  io.dropwizard.metrics
+  metrics-core
+  3.2.4
+
+
+  org.openjdk.jmh
+  jmh-core
+  1.19
+
+
+  org.openjdk.jmh
+  jmh-generator-annprocess
+  1.19
+
+
+  com.google.code.findbugs
+  findbugs
+  3.0.1
+  provided
+
+
+  junit
+  junit
+  test
+
+
+  org.apache.hadoop
+  hadoop-common
+  test
+  test-jar
+
+
+  org.apache.hadoop
+  hadoop-ozone-integration-test
+  test
+  test-jar
+
+
+  junit
+  junit
 
 Review comment:
   Just noticed this in Maven output: duplicate dependency on `junit:junit` 
(lines 100-104 and 117-121).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the visibility with Ozone Insight tool

2019-08-15 Thread GitBox
adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the 
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314292807
 
 

 ##
 File path: 
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/LogSubcommand.java
 ##
 @@ -0,0 +1,142 @@
+package org.apache.hadoop.ozone.insight;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.HttpClientBuilder;
+import picocli.CommandLine;
+
+/**
+ * Subcommand to display log.
+ */
+@CommandLine.Command(
+name = "log",
+aliases = "logs",
+description = "Show log4j events related to the insight point",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class LogSubcommand extends BaseInsightSubcommand
+implements Callable {
+
+  @CommandLine.Parameters(description = "Name of the insight point (use list "
+  + "to check the available options)")
+  private String insightName;
+
+  @CommandLine.Option(names = "-v", description = "Enable verbose mode to "
+  + "show more information / detailed message")
+  private boolean verbose;
+
+  @CommandLine.Parameters(defaultValue = "")
+  private String selection;
 
 Review comment:
   It seems only one of `insightName` and `selection` is needed.
   
* `insightName` is not used currently, but has description
* `selection` is not documented


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the visibility with Ozone Insight tool

2019-08-15 Thread GitBox
adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the 
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314290947
 
 

 ##
 File path: 
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/om/KeyManagerInsight.java
 ##
 @@ -0,0 +1,61 @@
+package org.apache.hadoop.ozone.insight.om;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.ozone.insight.BaseInsightPoint;
+import org.apache.hadoop.ozone.insight.Component.Type;
+import org.apache.hadoop.ozone.insight.LoggerSource;
+import org.apache.hadoop.ozone.insight.MetricDisplay;
+import org.apache.hadoop.ozone.insight.MetricGroupDisplay;
+import org.apache.hadoop.ozone.om.KeyManagerImpl;
+
+/**
+ * Insight implementation for the key management related operations.
+ */
+public class KeyManagerInsight extends BaseInsightPoint {
+
+  @Override
+  public List getMetrics() {
+List display = new ArrayList<>();
+
+MetricGroupDisplay state =
+new MetricGroupDisplay(Type.OM, "Key related metrics");
+state
+.addMetrics(new MetricDisplay("Number of keys", 
"om_metrics_num_keys"));
+state.addMetrics(new MetricDisplay("Number of key operations",
+"om_metrics_num_key_ops"));
+
+display.add(state);
+
+MetricGroupDisplay key =
+new MetricGroupDisplay(Type.OM, "Key operation stats");
+for (String operation : new String[] {"allocate", "commit", "lookup",
+"list", "delete"}) {
+  key.addMetrics(new MetricDisplay(
+  "Number of key " + operation + "s (failure + success)",
+  "om_metrics_num_key_" + operation));
+  key.addMetrics(
+  new MetricDisplay("Number of failed key " + operation + "s",
+  "om_metrics_num_key_" + operation + "_fails"));
+}
+display.add(key);
+
+return display;
+  }
+
+  @Override
+  public List getRelatedLoggers(boolean verbose) {
+List loggers = new ArrayList<>();
+loggers.add(
+new LoggerSource(Type.SCM, KeyManagerImpl.class,
 
 Review comment:
   I think it should be:
   
   ```suggestion
   new LoggerSource(Type.OM, KeyManagerImpl.class,
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the visibility with Ozone Insight tool

2019-08-15 Thread GitBox
adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the 
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314483408
 
 

 ##
 File path: 
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/ConfigurationSubCommand.java
 ##
 @@ -0,0 +1,70 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import picocli.CommandLine;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Subcommand to show configuration values/documentation.
+ */
+@CommandLine.Command(
+name = "config",
+description = "Show configuration for a specific subcomponents",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ConfigurationSubCommand extends BaseInsightSubcommand
+implements Callable {
+
+  @CommandLine.Parameters(defaultValue = "")
+  private String selection;
+
+  @Override
+  public Void call() throws Exception {
+InsightPoint insight =
+getInsight(getInsightCommand().createOzoneConfiguration(), selection);
+System.out.println(
+"Configuration for `" + selection + "` (" + insight.getDescription()
++ ")");
+System.out.println();
+for (Class clazz : insight.getConfigurationClasses()) {
+  showConfig(clazz);
+
+}
+return null;
+  }
+
+  private void showConfig(Class clazz) {
+OzoneConfiguration conf = new OzoneConfiguration();
+conf.addResource("http://localhost:9876/conf;);
 
 Review comment:
   Is this SCM-specific?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the visibility with Ozone Insight tool

2019-08-15 Thread GitBox
adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the 
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314274453
 
 

 ##
 File path: 
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ScmBlockLocationProtocolServerSideTranslatorPB.java
 ##
 @@ -97,15 +96,45 @@ public SCMBlockLocationResponse send(RpcController 
controller,
   SCMBlockLocationRequest request) throws ServiceException {
 String traceId = request.getTraceID();
 
+if (LOG.isTraceEnabled()) {
+  LOG.trace("BlockLocationProtocol {} request is received: 
{}",
+  request.getCmdType().toString(),
+  request.toString().replaceAll("\n", "n"));
+
+} else if (LOG.isDebugEnabled()) {
+  LOG.debug("BlockLocationProtocol {} request is received",
+  request.getCmdType().toString());
+}
+
+protocolMessageMetrics.increment(request.getCmdType());
+
+try (Scope scope = TracingUtil
+.importAndCreateScope(
+"ScmBlockLocationProtocol." + request.getCmdType(),
+request.getTraceID())) {
+  SCMBlockLocationResponse response =
+  processMessage(request, traceId);
+
+  if (LOG.isTraceEnabled()) {
+LOG.trace(
+"BlockLocationProtocol {} request is processed. Response: "
++ "{}",
+request.getCmdType().toString(),
+request.toString().replaceAll("\n", "n"));
 
 Review comment:
   ```suggestion
   response.toString().replaceAll("\n", "n"));
   ```
   
   Although response is already logged in `processMessage()`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the visibility with Ozone Insight tool

2019-08-15 Thread GitBox
adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the 
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314293659
 
 

 ##
 File path: 
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/MetricsSubCommand.java
 ##
 @@ -0,0 +1,114 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.HttpClientBuilder;
+import picocli.CommandLine;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Command line interface to show metrics for a specific component.
+ */
+@CommandLine.Command(
+name = "metrics",
+aliases = "metric",
+description = "Show available metrics.",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class MetricsSubCommand extends BaseInsightSubcommand
+implements Callable {
+
+  @CommandLine.Parameters(defaultValue = "")
 
 Review comment:
   `defaultValue = ""` prevents help from being shown for incomplete command 
(`ozone insight metrics`).  Instead it gives `No such component` error.
   
   Same for `ConfigurationSubCommand`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[GitHub] [hadoop] adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the visibility with Ozone Insight tool

2019-08-15 Thread GitBox
adoroszlai commented on a change in pull request #1255: HDDS-1935. Improve the 
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314484032
 
 

 ##
 File path: 
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/List.java
 ##
 @@ -0,0 +1,38 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import picocli.CommandLine;
+
+import java.util.Map;
+import java.util.concurrent.Callable;
+
+/**
+ * Subcommand to list of the available insight points.
+ */
+@CommandLine.Command(
+name = "list",
+description = "Show available insight points.",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class List extends BaseInsightSubcommand implements Callable {
+
+  @CommandLine.Parameters(defaultValue = "")
+  private String selection;
 
 Review comment:
   Do you plan to use this parameter, eg. to filter available insight point 
list?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org