[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772575#comment-16772575 ] Hudson commented on HDDS-1130: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16004 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16004/]) HDDS-1130. Make BenchMarkBlockManager multi-threaded. Contributed by (yqlin: rev 1d30fd94c6430492ce2f92883117eff56094eec0) * (edit) hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkBlockManager.java > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch, > HDDS-1130.003.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772561#comment-16772561 ] Yiqun Lin commented on HDDS-1130: - LGTM, +1. Committing. > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch, > HDDS-1130.003.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772083#comment-16772083 ] Hadoop QA commented on HDDS-1130: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 48s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 55s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 21s{color} | {color:red} tools in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 58m 54s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.freon.TestFreonWithDatanodeFastRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDDS-1130 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12959262/HDDS-1130.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 45f5e8d3888b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1e0ae6e | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDDS-Build/2308/artifact/out/patch-unit-hadoop-ozone_tools.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDDS-Build/2308/testReport/ | | Max. process+thread count | 2466 (vs. ulimit of 1) | | modules | C: hadoop-ozone/tools U: hadoop-ozone/tools | | Console output | https://builds.apache.org/job/PreCommit-HDDS-Build/2308/console |
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772008#comment-16772008 ] Lokesh Jain commented on HDDS-1130: --- [~linyiqun] Thanks for the suggestion! v3 patch fixes the findbugs and checkstyle issues. > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch, > HDDS-1130.003.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771808#comment-16771808 ] Yiqun Lin commented on HDDS-1130: - [~ljain], I'm afraid that findbugs issues still exist. One solution way is making {{initialize}} method as a static method as well. BTW, we might rename {{numPipelines}} to another name, it duplicates the name parameter in {{addPipelines}} mehtod. Then it leads the checkstyle warning. > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771784#comment-16771784 ] Hadoop QA commented on HDDS-1130: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 19s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 12s{color} | {color:orange} hadoop-ozone/tools: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 45s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 48s{color} | {color:red} hadoop-ozone/tools generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 18s{color} | {color:red} tools in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 54m 41s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-ozone/tools | | | Write to static field org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.blockManager from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:[line 120] | | | Write to static field org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.pipelineManager from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:[line 115] | | | Write to static field org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.scm from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771751#comment-16771751 ] Lokesh Jain commented on HDDS-1130: --- [~linyiqun] Thanks for reviewing the patch! v2 patch addresses your comments. | We can use {{GenericTestUtils#waitFor}} to replace thread sleep: The GenericTestUtils and other test directory files are not accessible here. The findbugs are being reported because of recent changes. I need the fields to be static for multi-threaded analysis. > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch, HDDS-1130.002.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771614#comment-16771614 ] Hadoop QA commented on HDDS-1130: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 37s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 45s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s{color} | {color:red} hadoop-ozone/tools generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 16s{color} | {color:red} tools in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 55m 24s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-ozone/tools | | | Write to static field org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.blockManager from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:[line 110] | | | Write to static field org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.pipelineManager from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:[line 105] | | | Write to static field org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.scm from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:from instance method org.apache.hadoop.ozone.genesis.BenchMarkBlockManager.initialize() At BenchMarkBlockManager.java:[line 104] | \\ \\ || Subsystem || Report/Notes || | Docker |
[jira] [Commented] (HDDS-1130) Make BenchMarkBlockManager multi-threaded
[ https://issues.apache.org/jira/browse/HDDS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771596#comment-16771596 ] Yiqun Lin commented on HDDS-1130: - The patch almost looks good to me, two minor comments: * Can we define a static variable {{NUM_PIPELINES}} to replace hard-coded value {{10}}? * We can use {{GenericTestUtils#waitFor}} to replace thread sleep: {noformat} GenericTestUtils.waitFor(() -> { return !blockManager.isScmInChillMode(); }, 100, 6); {noformat} > Make BenchMarkBlockManager multi-threaded > - > > Key: HDDS-1130 > URL: https://issues.apache.org/jira/browse/HDDS-1130 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1130.001.patch > > > Currently BenchMarkBlockManager is run by a single thread. We can make it > multi-threaded in order to have a better understanding of allocateBlock call > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org