[
https://issues.apache.org/jira/browse/HDDS-2325?focusedWorklogId=330490&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-330490
]
ASF GitHub Bot logged work on HDDS-2325:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Oct/19 13:05
Start Date: 18/Oct/19 13:05
Worklog Time Spent: 10m
Work Description: elek commented on pull request #60: HDDS-2325.
BenchMarkDatanodeDispatcher genesis test is failing with NPE
URL: https://github.com/apache/hadoop-ozone/pull/60
## What changes were proposed in this pull request?
TLDR; Fix BenchmarkDatanodeDispatcher.writeChunk test.
Genesis is a microbenchmark tool for Ozone based on JMH
(https://openjdk.java.net/projects/code-tools/jmh/).
Due to the recent Datanode changes the BenchMarkDatanodeDispatcher is
failing with NPE:
```
java.lang.NullPointerException
at
org.apache.hadoop.ozone.container.common.interfaces.Handler.<init>(Handler.java:69)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.<init>(KeyValueHandler.java:114)
at
org.apache.hadoop.ozone.container.common.interfaces.Handler.getHandlerForContainerType(Handler.java:78)
at
org.apache.hadoop.ozone.genesis.BenchMarkDatanodeDispatcher.initialize(BenchMarkDatanodeDispatcher.java:115)
at
org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest._jmh_tryInit_f_benchmarkdatanodedispatcher0_G(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:438)
at
org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest.createContainer_Throughput(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
at
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
And this is the just the biggest problem there are a few other problems. I
propose the following fixes:
**fix 1**: NPE is thrown because the 'context' object is required by
KeyValueHandler/Handler classes.
In fact the context is not required, we need two functionalities/info from
the context: the ability to send icr (IncrementalContainerReport) and the ID of
the datanode.
Law of Demeter principle suggests to have only the minimum required
information from other classes.
For example instead of having context but using only
context.getParent().getDatanodeDetails().getUuidString() we can have only the
UUID string which makes more easy to test (unit and benchmark) the
Handler/KeyValueHandler.
This is the biggest (but still small change) in this patch: I started to use
the datanodeId and an icrSender instead of having the full context.
**fix 2,3**: There were a few other problems. The scmId was missing if the
writeChunk was called from Benchmark and and the Checksum was also missing.
**fix 4**: I also had a few other problems: very huge containers are used
(default 5G) and as the benchmark starts with creating 100 containers it
requires 500G space by default. I adjusted the container size to make it
possible to run on local machine.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2325
## How was this patch tested?
```
./ozone genesis -benchmark=BenchMarkDatanodeDispatcher.writeChunk
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 330490)
Remaining Estimate: 0h
Time Spent: 10m
> BenchMarkDatanodeDispatcher genesis test is failing with NPE
> ------------------------------------------------------------
>
> Key: HDDS-2325
> URL: https://issues.apache.org/jira/browse/HDDS-2325
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Reporter: Marton Elek
> Assignee: Marton Elek
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ## What changes were proposed in this pull request?
> Genesis is a microbenchmark tool for Ozone based on JMH
> ([https://openjdk.java.net/projects/code-tools/jmh/).]
>
> Due to the recent Datanode changes the BenchMarkDatanodeDispatcher is failing
> with NPE:
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.interfaces.Handler.<init>(Handler.java:69)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.<init>(KeyValueHandler.java:114)
> at
> org.apache.hadoop.ozone.container.common.interfaces.Handler.getHandlerForContainerType(Handler.java:78)
> at
> org.apache.hadoop.ozone.genesis.BenchMarkDatanodeDispatcher.initialize(BenchMarkDatanodeDispatcher.java:115)
> at
> org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest._jmh_tryInit_f_benchmarkdatanodedispatcher0_G(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:438)
> at
> org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest.createContainer_Throughput(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
> at
> org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> And this is the just the biggest problem there are a few other problems. I
> propose the following fixes:
> *fix 1*: NPE is thrown because the 'context' object is required by
> KeyValueHandler/Handler classes.
> In fact the context is not required, we need two functionalities/info from
> the context: the ability to send icr (IncrementalContainerReport) and the ID
> of the datanode.
> Law of Demeter principle suggests to have only the minimum required
> information from other classes.
> For example instead of having context but using only
> context.getParent().getDatanodeDetails().getUuidString() we can have only the
> UUID string which makes more easy to test (unit and benchmark) the
> Handler/KeyValueHandler.
> This is the biggest (but still small change) in this patch: I started to use
> the datanodeId and an icrSender instead of having the full context.
> *fix 2,3:* There were a few other problems. The scmId was missing if the
> writeChunk was called from Benchmark and and the Checksum was also missing.
> *fix 4:* I also had a few other problems: very huge containers are used
> (default 5G) and as the benchmark starts with creating 100 containers it
> requires 500G space by default. I adjusted the container size to make it
> possible to run on local machine.
>
> ## How this patch can be tested?
> {code:java}
> ./ozone genesis -benchmark=BenchMarkDatanodeDispatcher.writeChunk{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
