[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769911#comment-16769911
]
Hudson commented on HDFS-14258:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15981 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15981/])
HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer (inigoiri:
rev dde0ab55aadcf7c9cf71dbe36d90e97da6bc9498)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch, HDFS-14258.8.patch, HDFS-14258.9.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769903#comment-16769903
]
Íñigo Goiri commented on HDFS-14258:
Thanks [~belugabehr] for the patch and dealing with the review.
Commited to trunk.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch, HDFS-14258.8.patch, HDFS-14258.9.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769879#comment-16769879
]
BELUGA BEHR commented on HDFS-14258:
It took me several tries, even locally, but I did eventually get success:
{code}
user@apache-dev:~/hadoop/hadoop$ mvn clean test
-Dtest=TestNameNodeMetadataConsistency,TestDataNodeLifeline,TestDataNodeVolumeFailure,TestUnderReplicatedBlocks,TestJournalNodeSync,TestDFSClientRetries,TestBPOfferService
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 13:39 min
[INFO] Finished at: 2019-02-15T18:50:50-05:00
[INFO] Final Memory: 398M/1516M
[INFO]
{code}
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch, HDFS-14258.8.patch, HDFS-14258.9.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769817#comment-16769817
]
Íñigo Goiri commented on HDFS-14258:
We will ignore the checkstyle and I believe the failed unit tests are unrelated.
To be safe [~belugabehr] do you mind verifying the unit tests?
Previous runs don't show the issue and I don't see anything that would trigger
this.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch, HDFS-14258.8.patch, HDFS-14258.9.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769795#comment-16769795
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
478 unchanged - 4 fixed = 478 total (was 482) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 168 unchanged - 7 fixed = 172 total (was 175) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958924/HDFS-14258.9.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 52b184780f68 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d10444e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs |
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769678#comment-16769678
]
Íñigo Goiri commented on HDFS-14258:
[^HDFS-14258.9.patch] LGTM.
+1
I'll commit once Yetus comes back.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch, HDFS-14258.8.patch, HDFS-14258.9.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768834#comment-16768834
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
478 unchanged - 4 fixed = 478 total (was 482) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 168 unchanged - 7 fixed = 172 total (was 175) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958788/HDFS-14258.8.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5fa1567d1015 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 64f28f9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26226/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26226/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768775#comment-16768775
]
Íñigo Goiri commented on HDFS-14258:
To avoid the 30 seconds, can we do some setter VisibleForTesting or equivalent?
>From previous JRIAs, I think WhiteBox is out of the question here.
What about using exception rules with expect or {{LambdaTestUtils#intercept}}.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch, HDFS-14258.8.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768599#comment-16768599
]
Íñigo Goiri commented on HDFS-14258:
OK, let's go with the Math.abs() approach.
Any chance we can add coverage to catch "Could not increase concurrent moves
thread count"?
BTW, shouldn't the message be the opposite?
I'd say the way to return false is if we can not decrease the number of
concurrent moves; am I missing something?
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766375#comment-16766375
]
BELUGA BEHR commented on HDFS-14258:
[~elgoiri] Ya, the comment formats don't bother me at all, both are valid. One
marks the sections, one marks the line-by-line. Also, I think sprinkling a
{{-1}} in there does not help code readability.
I can't stop you from making those changes on commit, but please consider the
latest patch for inclusion.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766357#comment-16766357
]
Íñigo Goiri commented on HDFS-14258:
{quote}
I am sorry. I do not understand this comment. Are you referring to Simulate
grabbing 2 threads ?? If so, this is the appropriate comment format.
{quote}
I'm referring to {{TestDataNodeReconfiguration#234}} and
{{TestDataNodeReconfiguration#249}} for example.
{quote}
I think it is more clear calling Math.abs than to introduce a magic number and
assuming that everyone understands basic math I'm not sure what about this you
do not like.
{quote}
I would personally go for:
{code}
final int delta = this.maxThreads - newMaxThreads;
LOG.debug("Change concurrent thread count to {} from {}", newMaxThreads,
this.maxThreads);
if (delta == 0) {
return true;
}
if (delta < 0) {
LOG.debug("Adding thread capacity: {}", -1*delta);
this.semaphore.release(-1*delta);
this.maxThreads = newMaxThreads;
return true;
}
...
boolean acquired = this.semaphore.tryAcquire(-1 * delta, duration,
TimeUnit.SECONDS);
{code}
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766322#comment-16766322
]
BELUGA BEHR commented on HDFS-14258:
[~elgoiri]
{quote}
Just one minor comment: you added some javadoc comments in the middle to mark
the different phases, we should make them regular comments.
{quote}
I am sorry. I do not understand this comment. Are you referring to {{Simulate
grabbing 2 threads}} ?? If so, this is the appropriate comment format.
I think it is more clear calling {{Math.abs}} than to introduce a magic number
and assuming that everyone understands basic math :) I'm not sure what about
this you do not like.
I hope you will consider accepting the latest patch. I very much appreciate
all your feedback and time.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766307#comment-16766307
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
478 unchanged - 4 fixed = 478 total (was 482) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 168 unchanged - 7 fixed = 172 total (was 175) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958409/HDFS-14258.7.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6f906b957438 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 20b92cd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26193/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26193/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766303#comment-16766303
]
Íñigo Goiri commented on HDFS-14258:
Thanks for the unit tests.
Just one minor comment: you added some javadoc comments in the middle to mark
the different phases, we should make them regular comments.
{quote}
And the Math.abs is required. When doing the deltas... if DN is reconfigured
from 50 threads to 45 threads, that is a change of -5. To pull five "tickets"
from the semaphore, to decrease the number of threads that can run, the code
must request to acquire +5 ticket.
{quote}
I got the condition the other way.
So here we will always have a negative number (117 and 120 take care of this).
Does it make sense to just do -1 * delta? Or maybe reverse delta and filter the
0 and negative case?
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch,
> HDFS-14258.6.patch, HDFS-14258.7.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765461#comment-16765461
]
Íñigo Goiri commented on HDFS-14258:
OK, if we change this behavior, I think we need to test it.
Agreed on 30 seconds extra time being too much.
Having a configuration parameter would be the right thing but I agree with you
that another knob to ignore is bad.
What about having a setter and just tune it for the test?
Minor comment: I think we can get rid of the {{Math.abs()}} for delta in
{{setMaxConcurrentMovers()}} as we are filtering the bad cases.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch, HDFS-14258.6.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765426#comment-16765426
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
478 unchanged - 4 fixed = 478 total (was 482) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 168 unchanged - 7 fixed = 172 total (was 175) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958291/HDFS-14258.6.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 83e742463dda 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5c10630 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26187/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765291#comment-16765291
]
Íñigo Goiri commented on HDFS-14258:
The checkstyle should be fine; the switch is all messed up.
My main concern at this point is the change of behavior.
Right now we have {{DEFAULT_RECONFIGURE_WAIT}} (which should have a comment
saying is seconds).
I think we can achieve the old behavior by setting it to a very large number.
Actually we should handle special values like 0 or MAX_VALUE to achieve special
behaviors.
I'm still on the fence with not making this a configuration; in any case, this
should be marked as changing behavior.
As a minor nit, we could do {{waitAllPeers(2, TimeUnit.SECONDS);}}.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765227#comment-16765227
]
BELUGA BEHR commented on HDFS-14258:
[~elgoiri] I'm attaching a patch again to see if it can get past the flaky
tests.
There are some checkstyle errors reported, but I ask that you please ignore
these. It is complaining about line indentation, however the entire method has
incorrect indentation so if I correct it in this patch, it will look very out
of place with the rest of the code.
Please consider for inclusion into the project. Thanks.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch, HDFS-14258.4.patch, HDFS-14258.5.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763177#comment-16763177
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
477 unchanged - 4 fixed = 477 total (was 481) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 168 unchanged - 7 fixed = 172 total (was 175) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
34s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957963/HDFS-14258.4.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f9374d26a522 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 75e8441 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26173/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763022#comment-16763022
]
Íñigo Goiri commented on HDFS-14258:
{quote}
In regards to a dead-lock, the noPeers Condition is associated with the lock
object. When waiting on that condition, the thread gives up the lock and only
becomes active once the Condition has been signaled and has re-acquired the
lock.
{quote}
As I said, fancy stuff. Leveraging the sync in Java makes the code much
cleaner. Let's go for it.
Regarding code coverage, we might be able to tune the parameters to make it
faster than 30 seconds.
Actually, this sounds like the kind of thing that should be configurable
actually.
One could even get the old behavior tuning this parameter.
I have to go through the tests but it wouldn't hurt to add some concurrency
test here.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763009#comment-16763009
]
BELUGA BEHR commented on HDFS-14258:
Based on our discussion here, I'm working on another change. BRB :)
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763000#comment-16763000
]
BELUGA BEHR commented on HDFS-14258:
[~elgoiri] I ReadWrite lock adds even more overhead and it's not really worth
it here. The methods you mentioned are mostly for testing (in fact, I'm not
sure that {{getNumPeers()}} is even used anymore at all) and only hold the
lock a very short time. It's just not worth it. I'm pretty sure we don't want
multiple calls to {{sendOOBToPeers()}} happening at the same time; just one
thread sending an OOB at a time.
In regards to a dead-lock, the {{noPeers}} {{Condition}} is associated with the
{{lock}} object. When waiting on that condition, the thread gives up the lock
and only becomes active once the {{Condition}} has been signaled and has
re-acquired the lock.
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/Condition.html#awaitNanos-long-
Finally, for the unit test, I am a bit hesitant to want to add this test. The
code is not configurable (nor should we give users yet another nob to ignore),
so it has to sleep for 30 seconds and that just adds even more time to the unit
tests. You tell me though and I'll be happy to add one; it is pretty simple to
code up.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762973#comment-16762973
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
477 unchanged - 4 fixed = 477 total (was 481) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 19 unchanged - 5 fixed = 19 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957931/HDFS-14258.3.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ed5cf656137f 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a65aca2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26171/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26171/testReport/ |
| Max. process+thread count | 3602 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762960#comment-16762960
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
477 unchanged - 4 fixed = 477 total (was 481) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 19 unchanged - 5 fixed = 19 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957928/HDFS-14258.2.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e41d7d386dc1 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a65aca2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26170/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762937#comment-16762937
]
Íñigo Goiri commented on HDFS-14258:
{quote}
Which section of code are you speaking to specifically?
{quote}
Previously we had raw synchronized and everything locked, switching to a
ReentrantLock with semaphores and signaling is obviously fancier (and better).
Regarding the change of behavior I think is fine; we need to be a little more
careful with the review but is fine.
We may want to add some explicit unit test for this explaining the behavior
(i.e., the 30s cap).
What about using a read/write lock for lock and then allow reading concurrently
(e.g., {{getNumPeers()}}, {{sendOOBToPeers()}},{{getNumPeersXceiver()}},...)?
BTW, might we have a deadlock in {{waitAllPeers()}}? We are locking and waiting
for peers to not be empty but then the ways to make peers empty are locked too.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch,
> HDFS-14258.3.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762841#comment-16762841
]
BELUGA BEHR commented on HDFS-14258:
[~elgoiri] I am flattered that you call my work "fancy." :) Which section of
code are you speaking to specifically?
My intent was to remove the looping sleep calls and to use the latest tools
(concurrent package) to do this in a more action-driven way, instead of
constant polling.
I updated the unit tests to allow for a PASS. There is a subtle difference
with this proposal over how the current implementation works and this is why
the unit tests didn't pass (and I accidentally left out the updated version,
sorry).
In the unlikely scenario that a configuration change is made in the middle of
operation (re-configuration), the current code in trunk will simply update a
"max thread" variable to the new cap and exit. All new threads will fail to
create if the cap is lowered, and eventually, running threads will complete and
the total number of threads will drop below the cap.
In this new proposed implementation, the thread changing the cap value will
block for all threads currently queued to do work, and then continue blocking
until enough running threads complete that the number of threads is below the
new cap. If the number of threads does not come under the cap within 30s, it
gives up and a warning is logged. I personally prefer this setup because it is
more opaque to the operator. If I set the maximum number of threads to X, but
then I still see X+n threads running, because they have not yet completed their
work, I am going to be confused as to the reason for this. With this proposed
implementation, an operator will see that applying the new cap was not
successful, the reason for the failure, and will know why there are currently
more than X threads running.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch, HDFS-14258.2.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762232#comment-16762232
]
Íñigo Goiri commented on HDFS-14258:
As we are getting fancy here, why not using the read write lock and let
concurrent reads?
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-14258.1.patch
>
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762226#comment-16762226
]
Hadoop QA commented on HDFS-14258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new + 477
unchanged - 4 fixed = 477 total (was 481) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 11 unchanged - 4 fixed = 14 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeReconfiguration |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957821/HDFS-14258.1.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 16be3e34754a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d3de8e1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-14258) Introduce Java Concurrent Package To DataXceiverServer Class
[
https://issues.apache.org/jira/browse/HDFS-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762047#comment-16762047
]
BELUGA BEHR commented on HDFS-14258:
If we go this route, we can add a timeout to the {{acquire}} method and allow
the DataNode to wait a few seconds before giving up, as things currently stand,
the DN will give up immediately if there are no resources available. There is
no queuing.
> Introduce Java Concurrent Package To DataXceiverServer Class
>
>
> Key: HDFS-14258
> URL: https://issues.apache.org/jira/browse/HDFS-14258
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> * Use Java concurrent package to replace current facilities in
> {{DataXceiverServer}}.
> * A little bit of extra clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
