[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=303964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303964
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 30/Aug/19 00:07
Start Date: 30/Aug/19 00:07
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 303964)
Time Spent: 4h (was: 3h 50m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>default: 9863
>current: 9863
> The port number of the Ozone SCM block client service.
> >>> ozone.scm.block.client.address
>default: ${ozone.scm.client.address}
>current: scm
> The address of the Ozone SCM block client service. If not defined value of
> ozone.scm.client.address is used
> {code}
> Metrics can be retrieved from the prometheus entrypoint:
> {code}
> ozone insight metrics scm.protocol.block-location
> Metrics for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> RPC connections
> Open connections: 0
> Dropped connections: 0
> Received bytes: 0
> Sent bytes: 0
> RPC queue
> RPC average queue time: 0.0
> RPC call queue length: 0
> RPC performance
> RPC processing time average: 0.0
> Number of slow calls: 0
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[ https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=303590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303590 ] ASF GitHub Bot logged work on HDDS-1935: Author: ASF GitHub Bot Created on: 29/Aug/19 10:59 Start Date: 29/Aug/19 10:59 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool URL: https://github.com/apache/hadoop/pull/1255#issuecomment-526135834 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 68 | Maven dependency ordering for branch | | +1 | mvninstall | 592 | trunk passed | | +1 | compile | 380 | trunk passed | | +1 | checkstyle | 79 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 871 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 180 | trunk passed | | 0 | spotbugs | 449 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 668 | trunk passed | | -0 | patch | 499 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 40 | Maven dependency ordering for patch | | +1 | mvninstall | 566 | the patch passed | | +1 | compile | 463 | the patch passed | | +1 | javac | 463 | the patch passed | | +1 | checkstyle | 103 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 38 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 12 | The patch has no ill-formed XML file. | | +1 | shadedclient | 892 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 226 | the patch passed | | +1 | findbugs | 851 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 358 | hadoop-hdds in the patch passed. | | -1 | unit | 1971 | hadoop-ozone in the patch failed. | | +1 | asflicense | 55 | The patch does not generate ASF License warnings. | | | | 8718 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestContainerStateMachine | | | hadoop.hdds.scm.pipeline.TestSCMPipelineManager | | | hadoop.ozone.om.snapshot.TestOzoneManagerSnapshotProvider | | | hadoop.ozone.client.rpc.TestContainerStateMachineFailures | | | hadoop.ozone.om.TestOzoneManagerHA | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1255 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs | | uname | Linux 5bbd24f32b4f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c749f62 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/10/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/10/testReport/ | | Max. process+thread count | 4300 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/config hadoop-hdds/framework hadoop-hdds/server-scm hadoop-ozone hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/insight hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/10/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 303590)
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=303463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303463
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 29/Aug/19 08:30
Start Date: 29/Aug/19 08:30
Worklog Time Spent: 10m
Work Description: elek commented on issue #1255: HDDS-1935. Improve the
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#issuecomment-526083709
> Can you please rebase this patch? The patch is not applying cleanly to the
head of trunk.
ListSubCommand.java
Could be a camel case problem on osx. I think earlier the camele case usage
was not consistent. Base on github there are no rebase problem. Can you please
try to delete your local insight folder and retry?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 303463)
Time Spent: 3h 40m (was: 3.5h)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>default: 9863
>current: 9863
> The port number of the Ozone SCM block client service.
> >>> ozone.scm.block.client.address
>default: ${ozone.scm.client.address}
>current: scm
> The address of the Ozone SCM block client service. If not defined value of
> ozone.scm.client.address is used
> {code}
> Metrics can be retrieved from the prometheus entrypoint:
> {code}
> ozone insight metrics scm.protocol.block-location
>
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=303202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303202
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 28/Aug/19 20:05
Start Date: 28/Aug/19 20:05
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#issuecomment-525901246
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 0 | Docker mode activated. |
| -1 | patch | 14 | https://github.com/apache/hadoop/pull/1255 does not
apply to trunk. Rebase required? Wrong Branch? See
https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/1255 |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/9/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 303202)
Time Spent: 3.5h (was: 3h 20m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>default:
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=303052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303052
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 28/Aug/19 17:11
Start Date: 28/Aug/19 17:11
Worklog Time Spent: 10m
Work Description: anuengineer commented on issue #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#issuecomment-525837063
Can you please rebase this patch? The patch is not applying cleanly to the
head of trunk.
`ListSubCommand.java`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 303052)
Time Spent: 3h 20m (was: 3h 10m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>default: 9863
>current: 9863
> The port number of the Ozone SCM block client service.
> >>> ozone.scm.block.client.address
>default: ${ozone.scm.client.address}
>current: scm
> The address of the Ozone SCM block client service. If not defined value of
> ozone.scm.client.address is used
> {code}
> Metrics can be retrieved from the prometheus entrypoint:
> {code}
> ozone insight metrics scm.protocol.block-location
> Metrics for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> RPC connections
> Open connections: 0
> Dropped connections: 0
> Received bytes: 0
> Sent bytes: 0
> RPC queue
>
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=302975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302975
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 28/Aug/19 15:46
Start Date: 28/Aug/19 15:46
Worklog Time Spent: 10m
Work Description: elek commented on issue #1255: HDDS-1935. Improve the
visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#issuecomment-525804792
> This is a very useful addition @elek. Is there any documentation or slides
that I can look at to understand this more?
I think all the CLI parameters are well documented. But I will definitely
create new doc pages if the patch is accepted.
@anuengineer @arp7: Do you have any more comments? Can you please review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 302975)
Time Spent: 3h 10m (was: 3h)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>default: 9863
>current: 9863
> The port number of the Ozone SCM block client service.
> >>> ozone.scm.block.client.address
>default: ${ozone.scm.client.address}
>current: scm
> The address of the Ozone SCM block client service. If not defined value of
> ozone.scm.client.address is used
> {code}
> Metrics can be retrieved from the prometheus entrypoint:
> {code}
> ozone insight metrics scm.protocol.block-location
>
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[ https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=302573=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302573 ] ASF GitHub Bot logged work on HDDS-1935: Author: ASF GitHub Bot Created on: 28/Aug/19 06:46 Start Date: 28/Aug/19 06:46 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool URL: https://github.com/apache/hadoop/pull/1255#issuecomment-525607376 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 2 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 80 | Maven dependency ordering for branch | | +1 | mvninstall | 697 | trunk passed | | +1 | compile | 406 | trunk passed | | +1 | checkstyle | 98 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 808 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 189 | trunk passed | | 0 | spotbugs | 449 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 674 | trunk passed | | -0 | patch | 521 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 50 | Maven dependency ordering for patch | | +1 | mvninstall | 588 | the patch passed | | +1 | compile | 412 | the patch passed | | +1 | javac | 412 | the patch passed | | +1 | checkstyle | 94 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 32 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 8 | The patch has no ill-formed XML file. | | +1 | shadedclient | 682 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 193 | the patch passed | | -1 | findbugs | 440 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | +1 | unit | 299 | hadoop-hdds in the patch passed. | | -1 | unit | 2262 | hadoop-ozone in the patch failed. | | +1 | asflicense | 63 | The patch does not generate ASF License warnings. | | | | 8627 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestWatchForCommit | | | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1255 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs | | uname | Linux 252630c31851 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b1eee8b | | Default Java | 1.8.0_222 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/8/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/8/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/8/testReport/ | | Max. process+thread count | 5340 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/config hadoop-hdds/framework hadoop-hdds/server-scm hadoop-ozone hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/insight hadoop-ozone/ozone-manager hadoop-ozone/tools U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/8/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=301777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-301777
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 27/Aug/19 08:25
Start Date: 27/Aug/19 08:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#issuecomment-525196961
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 0 | Docker mode activated. |
| -1 | patch | 13 | https://github.com/apache/hadoop/pull/1255 does not
apply to trunk. Rebase required? Wrong Branch? See
https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/1255 |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/7/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 301777)
Time Spent: 2h 50m (was: 2h 40m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[ https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=299451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-299451 ] ASF GitHub Bot logged work on HDDS-1935: Author: ASF GitHub Bot Created on: 22/Aug/19 13:54 Start Date: 22/Aug/19 13:54 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool URL: https://github.com/apache/hadoop/pull/1255#issuecomment-523916696 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | shelldocs | 1 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for branch | | +1 | mvninstall | 607 | trunk passed | | +1 | compile | 375 | trunk passed | | +1 | checkstyle | 82 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 800 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 170 | trunk passed | | 0 | spotbugs | 444 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 648 | trunk passed | | -0 | patch | 490 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 36 | Maven dependency ordering for patch | | +1 | mvninstall | 545 | the patch passed | | +1 | compile | 389 | the patch passed | | +1 | javac | 389 | the patch passed | | -0 | checkstyle | 41 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 26 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 6 | The patch has no ill-formed XML file. | | +1 | shadedclient | 676 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 161 | the patch passed | | -1 | findbugs | 416 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | +1 | unit | 291 | hadoop-hdds in the patch passed. | | -1 | unit | 1714 | hadoop-ozone in the patch failed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 7559 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.scm.pipeline.TestSCMPipelineMetrics | | | hadoop.ozone.container.server.TestSecureContainerServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1255 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs | | uname | Linux 6741578cdaa4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ee7c261 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/6/artifact/out/diff-checkstyle-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/6/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/6/testReport/ | | Max. process+thread count | 5307 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/config hadoop-hdds/framework hadoop-hdds/server-scm hadoop-ozone hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/insight hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/6/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[ https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=299031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-299031 ] ASF GitHub Bot logged work on HDDS-1935: Author: ASF GitHub Bot Created on: 21/Aug/19 21:29 Start Date: 21/Aug/19 21:29 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool URL: https://github.com/apache/hadoop/pull/1255#issuecomment-523656816 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 2 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 74 | Maven dependency ordering for branch | | +1 | mvninstall | 657 | trunk passed | | +1 | compile | 388 | trunk passed | | +1 | checkstyle | 77 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 791 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 188 | trunk passed | | 0 | spotbugs | 464 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 704 | trunk passed | | -0 | patch | 513 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 44 | Maven dependency ordering for patch | | +1 | mvninstall | 577 | the patch passed | | +1 | compile | 393 | the patch passed | | +1 | javac | 393 | the patch passed | | -0 | checkstyle | 43 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 26 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 8 | The patch has no ill-formed XML file. | | +1 | shadedclient | 667 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 174 | the patch passed | | -1 | findbugs | 429 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | +1 | unit | 304 | hadoop-hdds in the patch passed. | | -1 | unit | 2388 | hadoop-ozone in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 8502 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.container.server.TestSecureContainerServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1255 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs | | uname | Linux 89c76896c261 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 217e748 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/5/artifact/out/diff-checkstyle-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/5/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/5/testReport/ | | Max. process+thread count | 4673 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/config hadoop-hdds/framework hadoop-hdds/server-scm hadoop-ozone hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/insight hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/5/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[ https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=298568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298568 ] ASF GitHub Bot logged work on HDDS-1935: Author: ASF GitHub Bot Created on: 21/Aug/19 10:06 Start Date: 21/Aug/19 10:06 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool URL: https://github.com/apache/hadoop/pull/1255#issuecomment-523390736 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 69 | Maven dependency ordering for branch | | +1 | mvninstall | 604 | trunk passed | | +1 | compile | 386 | trunk passed | | +1 | checkstyle | 75 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 783 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 168 | trunk passed | | 0 | spotbugs | 428 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 618 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 36 | Maven dependency ordering for patch | | +1 | mvninstall | 564 | the patch passed | | +1 | compile | 395 | the patch passed | | +1 | javac | 395 | the patch passed | | +1 | checkstyle | 72 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 26 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 7 | The patch has no ill-formed XML file. | | +1 | shadedclient | 620 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 162 | the patch passed | | -1 | findbugs | 420 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | +1 | unit | 310 | hadoop-hdds in the patch passed. | | -1 | unit | 2044 | hadoop-ozone in the patch failed. | | +1 | asflicense | 51 | The patch does not generate ASF License warnings. | | | | 7892 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.Test2WayCommitInRatis | | | hadoop.ozone.container.server.TestSecureContainerServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1255 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs | | uname | Linux 74081c66ba67 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 8aaf5e1 | | Default Java | 1.8.0_212 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/4/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/4/testReport/ | | Max. process+thread count | 5284 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/config hadoop-hdds/framework hadoop-hdds/server-scm hadoop-ozone hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/insight hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/4/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298568) Time Spent: 2h 20m (was: 2h 10m) >
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=298478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298478
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 21/Aug/19 06:59
Start Date: 21/Aug/19 06:59
Worklog Time Spent: 10m
Work Description: elek commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r316024166
##
File path: hadoop-ozone/insight/pom.xml
##
@@ -0,0 +1,137 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
+https://maven.apache.org/xsd/maven-4.0.0.xsd;>
+ 4.0.0
+
+org.apache.hadoop
+hadoop-ozone
+0.5.0-SNAPSHOT
+
+ hadoop-ozone-insight
+ 0.5.0-SNAPSHOT
+ Apache Hadoop Ozone Insight Tool
+ Apache Hadoop Ozone Insight Tool
+ jar
+
+
+
+ org.apache.hadoop
+ hadoop-ozone-ozone-manager
+
+
+ org.apache.hadoop
+ hadoop-ozone-common
+
+
+
+ org.apache.hadoop
+ hadoop-hdds-server-scm
+
+
+ org.apache.hadoop
+ hadoop-ozone-client
+
+
+ org.apache.hadoop
+ hadoop-ozone-filesystem
+
+
+ org.apache.hadoop
+ hadoop-hdds-server-framework
+
+
+ org.apache.hadoop
+ hadoop-common
+ compile
+
+
+ org.apache.hadoop
+ hadoop-hdfs
+ compile
+
+
+ com.sun.xml.bind
+ jaxb-core
+
+
+ javax.xml.bind
+ jaxb-api
+
+
+ javax.activation
+ activation
+
+
+ io.dropwizard.metrics
+ metrics-core
+ 3.2.4
+
+
+ org.openjdk.jmh
+ jmh-core
+ 1.19
+
+
+ org.openjdk.jmh
+ jmh-generator-annprocess
+ 1.19
+
+
+ com.google.code.findbugs
+ findbugs
+ 3.0.1
+ provided
+
+
+ junit
+ junit
+ test
+
+
+ org.apache.hadoop
+ hadoop-common
+ test
+ test-jar
+
+
+ org.apache.hadoop
+ hadoop-ozone-integration-test
+ test
+ test-jar
+
+
+ junit
+ junit
Review comment:
Thanks, it's removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 298478)
Time Spent: 2h 10m (was: 2h)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=298477=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298477
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 21/Aug/19 06:58
Start Date: 21/Aug/19 06:58
Worklog Time Spent: 10m
Work Description: elek commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r316023955
##
File path:
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/ConfigurationSubCommand.java
##
@@ -0,0 +1,70 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import picocli.CommandLine;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Subcommand to show configuration values/documentation.
+ */
+@CommandLine.Command(
+name = "config",
+description = "Show configuration for a specific subcomponents",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ConfigurationSubCommand extends BaseInsightSubcommand
+implements Callable {
+
+ @CommandLine.Parameters(defaultValue = "")
+ private String selection;
+
+ @Override
+ public Void call() throws Exception {
+InsightPoint insight =
+getInsight(getInsightCommand().createOzoneConfiguration(), selection);
+System.out.println(
+"Configuration for `" + selection + "` (" + insight.getDescription()
++ ")");
+System.out.println();
+for (Class clazz : insight.getConfigurationClasses()) {
+ showConfig(clazz);
+
+}
+return null;
+ }
+
+ private void showConfig(Class clazz) {
+OzoneConfiguration conf = new OzoneConfiguration();
+conf.addResource("http://localhost:9876/conf;);
Review comment:
1. We can use any of the servers but SCM seems to be a safe choice as it
should run all of the time
2. I modified it to use `getHost()` instead of the hard coded localhost name
here...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 298477)
Time Spent: 2h (was: 1h 50m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=298471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298471
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 21/Aug/19 06:50
Start Date: 21/Aug/19 06:50
Worklog Time Spent: 10m
Work Description: elek commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r316021395
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ScmBlockLocationProtocolServerSideTranslatorPB.java
##
@@ -97,15 +96,45 @@ public SCMBlockLocationResponse send(RpcController
controller,
SCMBlockLocationRequest request) throws ServiceException {
String traceId = request.getTraceID();
+if (LOG.isTraceEnabled()) {
+ LOG.trace("BlockLocationProtocol {} request is received:
{}",
+ request.getCmdType().toString(),
+ request.toString().replaceAll("\n", "n"));
+
+} else if (LOG.isDebugEnabled()) {
+ LOG.debug("BlockLocationProtocol {} request is received",
+ request.getCmdType().toString());
+}
+
+protocolMessageMetrics.increment(request.getCmdType());
+
+try (Scope scope = TracingUtil
+.importAndCreateScope(
+"ScmBlockLocationProtocol." + request.getCmdType(),
+request.getTraceID())) {
+ SCMBlockLocationResponse response =
+ processMessage(request, traceId);
+
+ if (LOG.isTraceEnabled()) {
+LOG.trace(
+"BlockLocationProtocol {} request is processed. Response: "
++ "{}",
+request.getCmdType().toString(),
+request.toString().replaceAll("\n", "n"));
Review comment:
Thanks, I removed the duplicated response log and committed the suggested
change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 298471)
Time Spent: 1h 50m (was: 1h 40m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=298467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298467
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 21/Aug/19 06:45
Start Date: 21/Aug/19 06:45
Worklog Time Spent: 10m
Work Description: elek commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r316019985
##
File path:
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/List.java
##
@@ -0,0 +1,38 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import picocli.CommandLine;
+
+import java.util.Map;
+import java.util.concurrent.Callable;
+
+/**
+ * Subcommand to list of the available insight points.
+ */
+@CommandLine.Command(
+name = "list",
+description = "Show available insight points.",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class List extends BaseInsightSubcommand implements Callable {
+
+ @CommandLine.Parameters(defaultValue = "")
+ private String selection;
Review comment:
Yes, originally I used a tree like insight point structure and it helped to
list only a subtree. When I switched to use a flat structure it was no longer
supported.
I fixed it and restored this functionality. `ozone insight log scm` can list
the scm related insight points from now...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 298467)
Time Spent: 1h 40m (was: 1.5h)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=296192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296192
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 16/Aug/19 09:39
Start Date: 16/Aug/19 09:39
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314651145
##
File path: hadoop-ozone/insight/pom.xml
##
@@ -0,0 +1,137 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
+https://maven.apache.org/xsd/maven-4.0.0.xsd;>
+ 4.0.0
+
+org.apache.hadoop
+hadoop-ozone
+0.5.0-SNAPSHOT
+
+ hadoop-ozone-insight
+ 0.5.0-SNAPSHOT
+ Apache Hadoop Ozone Insight Tool
+ Apache Hadoop Ozone Insight Tool
+ jar
+
+
+
+ org.apache.hadoop
+ hadoop-ozone-ozone-manager
+
+
+ org.apache.hadoop
+ hadoop-ozone-common
+
+
+
+ org.apache.hadoop
+ hadoop-hdds-server-scm
+
+
+ org.apache.hadoop
+ hadoop-ozone-client
+
+
+ org.apache.hadoop
+ hadoop-ozone-filesystem
+
+
+ org.apache.hadoop
+ hadoop-hdds-server-framework
+
+
+ org.apache.hadoop
+ hadoop-common
+ compile
+
+
+ org.apache.hadoop
+ hadoop-hdfs
+ compile
+
+
+ com.sun.xml.bind
+ jaxb-core
+
+
+ javax.xml.bind
+ jaxb-api
+
+
+ javax.activation
+ activation
+
+
+ io.dropwizard.metrics
+ metrics-core
+ 3.2.4
+
+
+ org.openjdk.jmh
+ jmh-core
+ 1.19
+
+
+ org.openjdk.jmh
+ jmh-generator-annprocess
+ 1.19
+
+
+ com.google.code.findbugs
+ findbugs
+ 3.0.1
+ provided
+
+
+ junit
+ junit
+ test
+
+
+ org.apache.hadoop
+ hadoop-common
+ test
+ test-jar
+
+
+ org.apache.hadoop
+ hadoop-ozone-integration-test
+ test
+ test-jar
+
+
+ junit
+ junit
Review comment:
Just noticed this in Maven output: duplicate dependency on `junit:junit`
(lines 100-104 and 117-121).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296192)
Time Spent: 1.5h (was: 1h 20m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.security
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=295747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295747
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 15/Aug/19 20:55
Start Date: 15/Aug/19 20:55
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314293659
##
File path:
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/MetricsSubCommand.java
##
@@ -0,0 +1,114 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.HttpClientBuilder;
+import picocli.CommandLine;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Command line interface to show metrics for a specific component.
+ */
+@CommandLine.Command(
+name = "metrics",
+aliases = "metric",
+description = "Show available metrics.",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class MetricsSubCommand extends BaseInsightSubcommand
+implements Callable {
+
+ @CommandLine.Parameters(defaultValue = "")
Review comment:
`defaultValue = ""` prevents help from being shown for incomplete command
(`ozone insight metrics`). Instead it gives `No such component` error.
Same for `ConfigurationSubCommand`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 295747)
Time Spent: 50m (was: 40m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=295751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295751
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 15/Aug/19 20:55
Start Date: 15/Aug/19 20:55
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314290947
##
File path:
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/om/KeyManagerInsight.java
##
@@ -0,0 +1,61 @@
+package org.apache.hadoop.ozone.insight.om;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.ozone.insight.BaseInsightPoint;
+import org.apache.hadoop.ozone.insight.Component.Type;
+import org.apache.hadoop.ozone.insight.LoggerSource;
+import org.apache.hadoop.ozone.insight.MetricDisplay;
+import org.apache.hadoop.ozone.insight.MetricGroupDisplay;
+import org.apache.hadoop.ozone.om.KeyManagerImpl;
+
+/**
+ * Insight implementation for the key management related operations.
+ */
+public class KeyManagerInsight extends BaseInsightPoint {
+
+ @Override
+ public List getMetrics() {
+List display = new ArrayList<>();
+
+MetricGroupDisplay state =
+new MetricGroupDisplay(Type.OM, "Key related metrics");
+state
+.addMetrics(new MetricDisplay("Number of keys",
"om_metrics_num_keys"));
+state.addMetrics(new MetricDisplay("Number of key operations",
+"om_metrics_num_key_ops"));
+
+display.add(state);
+
+MetricGroupDisplay key =
+new MetricGroupDisplay(Type.OM, "Key operation stats");
+for (String operation : new String[] {"allocate", "commit", "lookup",
+"list", "delete"}) {
+ key.addMetrics(new MetricDisplay(
+ "Number of key " + operation + "s (failure + success)",
+ "om_metrics_num_key_" + operation));
+ key.addMetrics(
+ new MetricDisplay("Number of failed key " + operation + "s",
+ "om_metrics_num_key_" + operation + "_fails"));
+}
+display.add(key);
+
+return display;
+ }
+
+ @Override
+ public List getRelatedLoggers(boolean verbose) {
+List loggers = new ArrayList<>();
+loggers.add(
+new LoggerSource(Type.SCM, KeyManagerImpl.class,
Review comment:
I think it should be:
```suggestion
new LoggerSource(Type.OM, KeyManagerImpl.class,
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 295751)
Time Spent: 1h 20m (was: 1h 10m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanode
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=295748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295748
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 15/Aug/19 20:55
Start Date: 15/Aug/19 20:55
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314274453
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ScmBlockLocationProtocolServerSideTranslatorPB.java
##
@@ -97,15 +96,45 @@ public SCMBlockLocationResponse send(RpcController
controller,
SCMBlockLocationRequest request) throws ServiceException {
String traceId = request.getTraceID();
+if (LOG.isTraceEnabled()) {
+ LOG.trace("BlockLocationProtocol {} request is received:
{}",
+ request.getCmdType().toString(),
+ request.toString().replaceAll("\n", "n"));
+
+} else if (LOG.isDebugEnabled()) {
+ LOG.debug("BlockLocationProtocol {} request is received",
+ request.getCmdType().toString());
+}
+
+protocolMessageMetrics.increment(request.getCmdType());
+
+try (Scope scope = TracingUtil
+.importAndCreateScope(
+"ScmBlockLocationProtocol." + request.getCmdType(),
+request.getTraceID())) {
+ SCMBlockLocationResponse response =
+ processMessage(request, traceId);
+
+ if (LOG.isTraceEnabled()) {
+LOG.trace(
+"BlockLocationProtocol {} request is processed. Response: "
++ "{}",
+request.getCmdType().toString(),
+request.toString().replaceAll("\n", "n"));
Review comment:
```suggestion
response.toString().replaceAll("\n", "n"));
```
Although response is already logged in `processMessage()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 295748)
Time Spent: 1h (was: 50m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=295752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295752
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 15/Aug/19 20:55
Start Date: 15/Aug/19 20:55
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314292807
##
File path:
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/LogSubcommand.java
##
@@ -0,0 +1,142 @@
+package org.apache.hadoop.ozone.insight;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.HttpClientBuilder;
+import picocli.CommandLine;
+
+/**
+ * Subcommand to display log.
+ */
+@CommandLine.Command(
+name = "log",
+aliases = "logs",
+description = "Show log4j events related to the insight point",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class LogSubcommand extends BaseInsightSubcommand
+implements Callable {
+
+ @CommandLine.Parameters(description = "Name of the insight point (use list "
+ + "to check the available options)")
+ private String insightName;
+
+ @CommandLine.Option(names = "-v", description = "Enable verbose mode to "
+ + "show more information / detailed message")
+ private boolean verbose;
+
+ @CommandLine.Parameters(defaultValue = "")
+ private String selection;
Review comment:
It seems only one of `insightName` and `selection` is needed.
* `insightName` is not used currently, but has description
* `selection` is not documented
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 295752)
Time Spent: 1h 20m (was: 1h 10m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=295750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295750
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 15/Aug/19 20:55
Start Date: 15/Aug/19 20:55
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314483408
##
File path:
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/ConfigurationSubCommand.java
##
@@ -0,0 +1,70 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.Config;
+import org.apache.hadoop.hdds.conf.ConfigGroup;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import picocli.CommandLine;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.Callable;
+
+/**
+ * Subcommand to show configuration values/documentation.
+ */
+@CommandLine.Command(
+name = "config",
+description = "Show configuration for a specific subcomponents",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ConfigurationSubCommand extends BaseInsightSubcommand
+implements Callable {
+
+ @CommandLine.Parameters(defaultValue = "")
+ private String selection;
+
+ @Override
+ public Void call() throws Exception {
+InsightPoint insight =
+getInsight(getInsightCommand().createOzoneConfiguration(), selection);
+System.out.println(
+"Configuration for `" + selection + "` (" + insight.getDescription()
++ ")");
+System.out.println();
+for (Class clazz : insight.getConfigurationClasses()) {
+ showConfig(clazz);
+
+}
+return null;
+ }
+
+ private void showConfig(Class clazz) {
+OzoneConfiguration conf = new OzoneConfiguration();
+conf.addResource("http://localhost:9876/conf;);
Review comment:
Is this SCM-specific?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 295750)
Time Spent: 1h 10m (was: 1h)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=295749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295749
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 15/Aug/19 20:55
Start Date: 15/Aug/19 20:55
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#discussion_r314484032
##
File path:
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/List.java
##
@@ -0,0 +1,38 @@
+package org.apache.hadoop.ozone.insight;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import picocli.CommandLine;
+
+import java.util.Map;
+import java.util.concurrent.Callable;
+
+/**
+ * Subcommand to list of the available insight points.
+ */
+@CommandLine.Command(
+name = "list",
+description = "Show available insight points.",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class List extends BaseInsightSubcommand implements Callable {
+
+ @CommandLine.Parameters(defaultValue = "")
+ private String selection;
Review comment:
Do you plan to use this parameter, eg. to filter available insight point
list?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 295749)
Time Spent: 1h 10m (was: 1h)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
>
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=294125=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294125
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 13/Aug/19 19:43
Start Date: 13/Aug/19 19:43
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on issue #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#issuecomment-520979879
This is a very useful addition @elek. Is there any documentation or slides
that I can look at to understand this more?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 294125)
Time Spent: 40m (was: 0.5h)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>default: 9863
>current: 9863
> The port number of the Ozone SCM block client service.
> >>> ozone.scm.block.client.address
>default: ${ozone.scm.client.address}
>current: scm
> The address of the Ozone SCM block client service. If not defined value of
> ozone.scm.client.address is used
> {code}
> Metrics can be retrieved from the prometheus entrypoint:
> {code}
> ozone insight metrics scm.protocol.block-location
> Metrics for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> RPC connections
> Open connections: 0
> Dropped connections: 0
> Received bytes: 0
> Sent bytes: 0
> RPC queue
> RPC
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=293340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293340
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 12/Aug/19 19:54
Start Date: 12/Aug/19 19:54
Worklog Time Spent: 10m
Work Description: anuengineer commented on issue #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255#issuecomment-520570841
Let us sync up some time. If I get an overview of the code layout, it will
be easier for me to review this. I really appreciate you doing this. Thank you
... I will sync with you when you are back
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 293340)
Time Spent: 0.5h (was: 20m)
> Improve the visibility with Ozone Insight tool
> --
>
> Key: HDDS-1935
> URL: https://issues.apache.org/jira/browse/HDDS-1935
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Visibility is a key aspect for the operation of any Ozone cluster. We need
> better visibility to improve correctnes and performance. While the
> distributed tracing is a good tool for improving the visibility of
> performance we have no powerful tool which can be used to check the internal
> state of the Ozone cluster and debug certain correctness issues.
> To improve the visibility of the internal components I propose to introduce a
> new command line application `ozone insight`.
> The new tool will show the selected metrics / logs / configuration for any of
> the internal components (like replication-manager, pipeline, etc.).
> For each insight points we can define the required logs and log levels,
> metrics and configuration and the tool can display only the component
> specific information during the debug.
> h2. Usage
> First we can check the available insight point:
> {code}
> bash-4.2$ ozone insight list
> Available insight points:
> scm.node-manager SCM Datanode management related
> information.
> scm.replica-manager SCM closed container replication
> manager
> scm.event-queue Information about the internal async
> event delivery
> scm.protocol.block-location SCM Block location protocol endpoint
> scm.protocol.container-location Planned insight point which is not yet
> implemented.
> scm.protocol.datanodePlanned insight point which is not yet
> implemented.
> scm.protocol.securityPlanned insight point which is not yet
> implemented.
> scm.http Planned insight point which is not yet
> implemented.
> om.key-manager OM Key Manager
> om.protocol.client Ozone Manager RPC endpoint
> om.http Planned insight point which is not yet
> implemented.
> datanode.pipeline[id]More information about one ratis
> datanode ring.
> datanode.rocksdb More information about one ratis
> datanode ring.
> s3g.http Planned insight point which is not yet
> implemented.
> {code}
> Insight points can define configuration, metrics and/or logs. Configuration
> can be displayed based on the configuration objects:
> {code}
> ozone insight config scm.protocol.block-location
> Configuration for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> >>> ozone.scm.block.client.bind.host
>default: 0.0.0.0
>current: 0.0.0.0
> The hostname or IP address used by the SCM block client endpoint to bind
> >>> ozone.scm.block.client.port
>default: 9863
>current: 9863
> The port number of the Ozone SCM block client service.
> >>> ozone.scm.block.client.address
>default: ${ozone.scm.client.address}
>current: scm
> The address of the Ozone SCM block client service. If not defined value of
> ozone.scm.client.address is used
> {code}
> Metrics can be retrieved from the prometheus entrypoint:
> {code}
> ozone insight metrics scm.protocol.block-location
> Metrics for `scm.protocol.block-location` (SCM Block location protocol
> endpoint)
> RPC connections
> Open connections: 0
>
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[ https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=291467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-291467 ] ASF GitHub Bot logged work on HDDS-1935: Author: ASF GitHub Bot Created on: 08/Aug/19 17:47 Start Date: 08/Aug/19 17:47 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool URL: https://github.com/apache/hadoop/pull/1255#issuecomment-519620436 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 53 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | shelldocs | 1 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 32 | Maven dependency ordering for branch | | +1 | mvninstall | 647 | trunk passed | | +1 | compile | 398 | trunk passed | | +1 | checkstyle | 79 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 760 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | trunk passed | | 0 | spotbugs | 415 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 610 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 37 | Maven dependency ordering for patch | | +1 | mvninstall | 536 | the patch passed | | +1 | compile | 369 | the patch passed | | +1 | javac | 369 | the patch passed | | +1 | checkstyle | 77 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 26 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 6 | The patch has no ill-formed XML file. | | +1 | shadedclient | 630 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 154 | the patch passed | | -1 | findbugs | 419 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | +1 | unit | 301 | hadoop-hdds in the patch passed. | | -1 | unit | 1977 | hadoop-ozone in the patch failed. | | +1 | asflicense | 47 | The patch does not generate ASF License warnings. | | | | 7760 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1255 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs | | uname | Linux 127495e05306 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 63161cf | | Default Java | 1.8.0_212 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/1/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/1/testReport/ | | Max. process+thread count | 5005 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/config hadoop-hdds/framework hadoop-hdds/server-scm hadoop-ozone hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/insight hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1255/1/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,
[jira] [Work logged] (HDDS-1935) Improve the visibility with Ozone Insight tool
[
https://issues.apache.org/jira/browse/HDDS-1935?focusedWorklogId=291262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-291262
]
ASF GitHub Bot logged work on HDDS-1935:
Author: ASF GitHub Bot
Created on: 08/Aug/19 13:39
Start Date: 08/Aug/19 13:39
Worklog Time Spent: 10m
Work Description: elek commented on pull request #1255: HDDS-1935.
Improve the visibility with Ozone Insight tool
URL: https://github.com/apache/hadoop/pull/1255
Visibility is a key aspect for the operation of any Ozone cluster. We need
better visibility to improve correctnes and performance. While the distributed
tracing is a good tool for improving the visibility of performance we have no
powerful tool which can be used to check the internal state of the Ozone
cluster and debug certain correctness issues.
To improve the visibility of the internal components I propose to introduce
a new command line application `ozone insight`.
The new tool will show the selected metrics / logs / configuration for any
of the internal components (like replication-manager, pipeline, etc.).
For each insight points we can define the required logs and log levels,
metrics and configuration and the tool can display only the component specific
information during the debug.
h2. Usage
First we can check the available insight point:
{code}
bash-4.2$ ozone insight list
Available insight points:
scm.node-manager SCM Datanode management related
information.
scm.replica-manager SCM closed container replication
manager
scm.event-queue Information about the internal async
event delivery
scm.protocol.block-location SCM Block location protocol endpoint
scm.protocol.container-location Planned insight point which is not
yet implemented.
scm.protocol.datanodePlanned insight point which is not
yet implemented.
scm.protocol.securityPlanned insight point which is not
yet implemented.
scm.http Planned insight point which is not
yet implemented.
om.key-manager OM Key Manager
om.protocol.client Ozone Manager RPC endpoint
om.http Planned insight point which is not
yet implemented.
datanode.pipeline[id]More information about one ratis
datanode ring.
datanode.rocksdb More information about one ratis
datanode ring.
s3g.http Planned insight point which is not
yet implemented.
{code}
Insight points can define configuration, metrics and/or logs. Configuration
can be displayed based on the configuration objects:
{code}
ozone insight config scm.protocol.block-location
Configuration for `scm.protocol.block-location` (SCM Block location protocol
endpoint)
>>> ozone.scm.block.client.bind.host
default: 0.0.0.0
current: 0.0.0.0
The hostname or IP address used by the SCM block client endpoint to bind
>>> ozone.scm.block.client.port
default: 9863
current: 9863
The port number of the Ozone SCM block client service.
>>> ozone.scm.block.client.address
default: ${ozone.scm.client.address}
current: scm
The address of the Ozone SCM block client service. If not defined value of
ozone.scm.client.address is used
{code}
Metrics can be retrieved from the prometheus entrypoint:
{code}
ozone insight metrics scm.protocol.block-location
Metrics for `scm.protocol.block-location` (SCM Block location protocol
endpoint)
RPC connections
Open connections: 0
Dropped connections: 0
Received bytes: 0
Sent bytes: 0
RPC queue
RPC average queue time: 0.0
RPC call queue length: 0
RPC performance
RPC processing time average: 0.0
Number of slow calls: 0
Message type counters
Number of AllocateScmBlock: 0
Number of DeleteScmKeyBlocks: 0
Number of GetScmInfo: 2
Number of SortDatanodes: 0
{code}
Log levels can be adjusted with the existing logLevel servlet and can be
collected / streamd via a simple logstream servlet:
{code}
ozone insight log scm.node-manager
[SCM] 2019-08-08 12:42:37,392
[DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager]
Processing node report from [datanode=ozone_datanode_1.ozone_default]
[SCM] 2019-08-08 12:43:37,392
[DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager]
Processing node report from [datanode=ozone_datanode_1.ozone_default]
[SCM] 2019-08-08 12:44:37,392
