[jira] [Commented] (HDFS-12716) 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available
[ https://issues.apache.org/jira/browse/HDFS-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525922#comment-16525922 ] Ranith Sardar commented on HDFS-12716: -- Hi [~brahmareddy] , I have updated the patch. Please review it once. Thank you. > 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes > to be available > - > > Key: HDFS-12716 > URL: https://issues.apache.org/jira/browse/HDFS-12716 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: usharani >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-12716.002.patch, HDFS-12716.patch > > > Currently 'dfs.datanode.failed.volumes.tolerated' supports number of > tolerated failed volumes to be mentioned. This configuration change requires > restart of datanode. Since datanode volumes can be changed dynamically, > keeping this configuration same for all may not be good idea. > Support 'dfs.datanode.failed.volumes.tolerated' to accept special > 'negative value 'x' to tolerate failures of upto "n-x" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12716) 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available
[ https://issues.apache.org/jira/browse/HDFS-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar updated HDFS-12716: - Attachment: HDFS-12716.002.patch > 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes > to be available > - > > Key: HDFS-12716 > URL: https://issues.apache.org/jira/browse/HDFS-12716 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: usharani >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-12716.002.patch, HDFS-12716.patch > > > Currently 'dfs.datanode.failed.volumes.tolerated' supports number of > tolerated failed volumes to be mentioned. This configuration change requires > restart of datanode. Since datanode volumes can be changed dynamically, > keeping this configuration same for all may not be good idea. > Support 'dfs.datanode.failed.volumes.tolerated' to accept special > 'negative value 'x' to tolerate failures of upto "n-x" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-198) Create AuditLogger mechanism to be used by OM, SCM and Datanode
[ https://issues.apache.org/jira/browse/HDDS-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-198: --- Status: Patch Available (was: In Progress) > Create AuditLogger mechanism to be used by OM, SCM and Datanode > --- > > Key: HDDS-198 > URL: https://issues.apache.org/jira/browse/HDDS-198 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: audit, log4j2 > Attachments: HDDS-198.001.patch > > > This Jira tracks the work to create a custom AuditLogger which can be used by > OM, SCM, Datanode for auditing read/write events. > The AuditLogger will be designed using log4j2 and leveraging the MarkerFilter > approach to be able to turn on/off audit of read/write events by simply > changing the log config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-198) Create AuditLogger mechanism to be used by OM, SCM and Datanode
[ https://issues.apache.org/jira/browse/HDDS-198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525889#comment-16525889 ] Dinesh Chitlangia commented on HDDS-198: [~anu] , [~xyao] - Please help to review patch and share feedback for revisions. Thanks! > Create AuditLogger mechanism to be used by OM, SCM and Datanode > --- > > Key: HDDS-198 > URL: https://issues.apache.org/jira/browse/HDDS-198 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: audit, log4j2 > Attachments: HDDS-198.001.patch > > > This Jira tracks the work to create a custom AuditLogger which can be used by > OM, SCM, Datanode for auditing read/write events. > The AuditLogger will be designed using log4j2 and leveraging the MarkerFilter > approach to be able to turn on/off audit of read/write events by simply > changing the log config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-187) Command status publisher for datanode
[
https://issues.apache.org/jira/browse/HDDS-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525887#comment-16525887
]
genericqa commented on HDDS-187:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDDS-187 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDDS-187 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929504/HDDS-187.00.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/379/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Command status publisher for datanode
> -
>
> Key: HDDS-187
> URL: https://issues.apache.org/jira/browse/HDDS-187
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.2.1
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-187.00.patch
>
>
> Currently SCM sends set of commands for DataNode. DataNode executes them via
> CommandHandler. This jira intends to create a Command status publisher which
> will return status of these commands back to the SCM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-198) Create AuditLogger mechanism to be used by OM, SCM and Datanode
[ https://issues.apache.org/jira/browse/HDDS-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-198: --- Attachment: HDDS-198.001.patch > Create AuditLogger mechanism to be used by OM, SCM and Datanode > --- > > Key: HDDS-198 > URL: https://issues.apache.org/jira/browse/HDDS-198 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: audit, log4j2 > Attachments: HDDS-198.001.patch > > > This Jira tracks the work to create a custom AuditLogger which can be used by > OM, SCM, Datanode for auditing read/write events. > The AuditLogger will be designed using log4j2 and leveraging the MarkerFilter > approach to be able to turn on/off audit of read/write events by simply > changing the log config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12976) Introduce ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525875#comment-16525875
]
Erik Krogen commented on HDFS-12976:
I think to pursue an approach like [~shv]'s, we need to define some new
protocol, e.g. {{HAStatusProtocol}}, which {{HAServiceProtocol}} and
{{ClientProtocol}} (and others?) both extend, and has just one method like
{{getServiceState()}}. By having a separate protocol we can define the ACLs
differently, and set up inheritance for proper casting.
> Introduce ObserverReadProxyProvider
> ---
>
> Key: HDFS-12976
> URL: https://issues.apache.org/jira/browse/HDFS-12976
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12976-HDFS-12943.000.patch,
> HDFS-12976-HDFS-12943.001.patch, HDFS-12976-HDFS-12943.002.patch,
> HDFS-12976-HDFS-12943.003.patch, HDFS-12976-HDFS-12943.004.patch,
> HDFS-12976-HDFS-12943.005.patch, HDFS-12976.WIP.patch
>
>
> {{StandbyReadProxyProvider}} should implement {{FailoverProxyProvider}}
> interface and be able to submit read requests to ANN and SBN(s).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-187) Command status publisher for datanode
[ https://issues.apache.org/jira/browse/HDDS-187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-187: Status: Patch Available (was: Open) > Command status publisher for datanode > - > > Key: HDDS-187 > URL: https://issues.apache.org/jira/browse/HDDS-187 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-187.00.patch > > > Currently SCM sends set of commands for DataNode. DataNode executes them via > CommandHandler. This jira intends to create a Command status publisher which > will return status of these commands back to the SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-198) Create AuditLogger mechanism to be used by OM, SCM and Datanode
[ https://issues.apache.org/jira/browse/HDDS-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-198: --- Description: This Jira tracks the work to create a custom AuditLogger which can be used by OM, SCM, Datanode for auditing read/write events. The AuditLogger will be designed using log4j2 and leveraging the MarkerFilter approach to be able to turn on/off audit of read/write events by simply changing the log config. was: This Jira tracks the work to create a custom AuditLogger which can be used by KSM, SCM, Datanode for auditing read/write events. The AuditLogger will be designed using log4j2 and leveraging the MarkerFilter approach to be able to turn on/off audit of read/write events by simply changing the log config. > Create AuditLogger mechanism to be used by OM, SCM and Datanode > --- > > Key: HDDS-198 > URL: https://issues.apache.org/jira/browse/HDDS-198 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: audit, log4j2 > > This Jira tracks the work to create a custom AuditLogger which can be used by > OM, SCM, Datanode for auditing read/write events. > The AuditLogger will be designed using log4j2 and leveraging the MarkerFilter > approach to be able to turn on/off audit of read/write events by simply > changing the log config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-198) Create AuditLogger mechanism to be used by OM, SCM and Datanode
[ https://issues.apache.org/jira/browse/HDDS-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-198: --- Summary: Create AuditLogger mechanism to be used by OM, SCM and Datanode (was: Create AuditLogger mechanism to be used by KSM, SCM and Datanode) > Create AuditLogger mechanism to be used by OM, SCM and Datanode > --- > > Key: HDDS-198 > URL: https://issues.apache.org/jira/browse/HDDS-198 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: audit, log4j2 > > This Jira tracks the work to create a custom AuditLogger which can be used by > KSM, SCM, Datanode for auditing read/write events. > The AuditLogger will be designed using log4j2 and leveraging the MarkerFilter > approach to be able to turn on/off audit of read/write events by simply > changing the log config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6525) FsShell supports HDFS TTL
[
https://issues.apache.org/jira/browse/HDFS-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525847#comment-16525847
]
genericqa commented on HDFS-6525:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
51s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m
21s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 21s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
54s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
18s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
9s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-6525 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12652123/HDFS-6525.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux f2c936d4b0a2 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8752a48 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
[jira] [Updated] (HDDS-187) Command status publisher for datanode
[ https://issues.apache.org/jira/browse/HDDS-187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-187: Attachment: HDDS-187.00.patch > Command status publisher for datanode > - > > Key: HDDS-187 > URL: https://issues.apache.org/jira/browse/HDDS-187 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-187.00.patch > > > Currently SCM sends set of commands for DataNode. DataNode executes them via > CommandHandler. This jira intends to create a Command status publisher which > will return status of these commands back to the SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-195) Create generic CommandWatcher utility
[
https://issues.apache.org/jira/browse/HDDS-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525841#comment-16525841
]
genericqa commented on HDDS-195:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} framework in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDDS-195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929498/HDDS-195.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fb59c48f22b8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8752a48 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/378/testReport/ |
| Max. process+thread count | 336 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/framework U: hadoop-hdds/framework |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/378/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create generic CommandWatcher utility
> -
>
> Key: HDDS-195
> URL: https://issues.apache.org/jira/browse/HDDS-195
> Project: Hadoop
[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525822#comment-16525822 ] Todd Lipcon commented on HDFS-13703: Different set of random tests failed this time. I am guessing these are just flakes. I think this is ready to commit. > Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit > > > Key: HDFS-13703 > URL: https://issues.apache.org/jira/browse/HDFS-13703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13703.patch, hdfs-13703.patch > > > The DFSClient creates a CorruptedBlocks object, which contains a HashMap, on > every read call. In most cases, a read will not hit any corrupted blocks, and > this hashmap is not used. It seems the JIT isn't smart enough to eliminate > this allocation. We would be better off avoiding it and only allocating in > the rare case when a corrupt block is hit. > Removing this allocation reduced CPU usage of a TeraValidate job by about 10%. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525820#comment-16525820 ] Todd Lipcon commented on HDFS-13702: The test failure seems unrelated. Given I have a +1 on this from Stack and only a =0 from [~ste...@apache.org] I'll plan to commit this tomorrow unless someone objects. I think we can continue the discussion about HTrace removal in HADOOP-15566. > HTrace hooks taking 10-15% CPU in DFS client when disabled > -- > > Key: HDFS-13702 > URL: https://issues.apache.org/jira/browse/HDFS-13702 > Project: Hadoop HDFS > Issue Type: Bug > Components: performance >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch > > > I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate > workload even when HTrace is disabled. This is because it stringifies several > integers. We should avoid all allocation and stringification when htrace is > disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13524) Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
[
https://issues.apache.org/jira/browse/HDFS-13524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525802#comment-16525802
]
genericqa commented on HDFS-13524:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13524 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929490/HDFS-13524.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux eed8eaadff1b 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8752a48 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24513/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24513/testReport/ |
| Max. process+thread count | 3612 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24513/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Commented] (HDDS-195) Create generic CommandWatcher utility
[ https://issues.apache.org/jira/browse/HDDS-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525800#comment-16525800 ] Elek, Marton commented on HDDS-195: --- Thank you very much the comments [~anu] 1. I agree. 2. True, I opened HDDS-201 3. Fixed. It's a constructor parameter from now. 4. Fixed. Thanks to catch it. > Create generic CommandWatcher utility > - > > Key: HDDS-195 > URL: https://issues.apache.org/jira/browse/HDDS-195 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-195.001.patch, HDDS-195.002.patch, > HDDS-195.003.patch, HDDS-195.004.patch > > > In some cases we need a class which can track the status of the outgoing > commands. > The commands should be resent after a while except a status message is > received about command completion. > On high level, we need a builder factory, which takes the following > parameters: > * (destination) event type and the payload of the command which should be > repeated. > * the ID of the command/event > * The event/topic of the completion messages. If an IdentifiableEventPayload > is received on this topic, the specific event is done and don't need to > resend it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-195) Create generic CommandWatcher utility
[ https://issues.apache.org/jira/browse/HDDS-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-195: -- Attachment: HDDS-195.004.patch > Create generic CommandWatcher utility > - > > Key: HDDS-195 > URL: https://issues.apache.org/jira/browse/HDDS-195 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-195.001.patch, HDDS-195.002.patch, > HDDS-195.003.patch, HDDS-195.004.patch > > > In some cases we need a class which can track the status of the outgoing > commands. > The commands should be resent after a while except a status message is > received about command completion. > On high level, we need a builder factory, which takes the following > parameters: > * (destination) event type and the payload of the command which should be > repeated. > * the ID of the command/event > * The event/topic of the completion messages. If an IdentifiableEventPayload > is received on this topic, the specific event is done and don't need to > resend it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-201) Add name for LeaseManager
Elek, Marton created HDDS-201: - Summary: Add name for LeaseManager Key: HDDS-201 URL: https://issues.apache.org/jira/browse/HDDS-201 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Elek, Marton During the review of HDDS-195 we realised that one server could have multiple LeaseManagers (for example one for the watchers one for the container creation). To make it easier to monitor it would be good to use some specific names for the release manager. This jira is about adding a new field (name) to the release manager which should be defined by a constructor parameter and should be required. It should be used in the name of the Threads and all the log message (Something like "Starting CommandWatcher LeasManager") -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13704) Support expiry time in AdlFileSystem
[ https://issues.apache.org/jira/browse/HDFS-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525792#comment-16525792 ] Íñigo Goiri commented on HDFS-13704: HDFS-6525 currently is defined within HDFS-6382 as supporting HDFS TTL. However, this could be general and be used in cases like ADLs. Here, we only need to capture the setXattr in AdlFileSystem and use the proper ADLS API if the xttar is something like user.expiry or user.ttl. > Support expiry time in AdlFileSystem > > > Key: HDFS-13704 > URL: https://issues.apache.org/jira/browse/HDFS-13704 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Anbang Hu >Priority: Major > > ADLS supports setting an expiration time for a file. > We can leverage Xattr in FileSystem to set the expiration time. > This could use the same xattr as HDFS-6382 and the interface from HDFS-6525. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13704) Support expiry time in AdlFileSystem
[ https://issues.apache.org/jira/browse/HDFS-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13704: --- Description: ADLS supports setting an expiration time for a file. We can leverage Xattr in FileSystem to set the expiration time. This could use the same xattr as HDFS-6382 and the interface from HDFS-6525. was: ADLS supports setting an expiration time for a file. We can leverage Xattr in FileSystem to set the expiration time. This could use the same xattr as HDFS-6382. > Support expiry time in AdlFileSystem > > > Key: HDFS-13704 > URL: https://issues.apache.org/jira/browse/HDFS-13704 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Anbang Hu >Priority: Major > > ADLS supports setting an expiration time for a file. > We can leverage Xattr in FileSystem to set the expiration time. > This could use the same xattr as HDFS-6382 and the interface from HDFS-6525. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13704) Support expiry time in AdlFileSystem
Íñigo Goiri created HDFS-13704: -- Summary: Support expiry time in AdlFileSystem Key: HDFS-13704 URL: https://issues.apache.org/jira/browse/HDFS-13704 Project: Hadoop HDFS Issue Type: Improvement Reporter: Íñigo Goiri Assignee: Anbang Hu ADLS supports setting an expiration time for a file. We can leverage Xattr in FileSystem to set the expiration time. This could use the same xattr as HDFS-6382. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12976) Introduce ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525770#comment-16525770
]
genericqa commented on HDFS-12976:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
15s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
18s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 29m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
5s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
44s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}240m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.currentProxyIndex;
locked 60% of time Unsynchronized access at
ConfiguredFailoverProxyProvider.java:60% of time Unsynchronized access at
ConfiguredFailoverProxyProvider.java:[line 181] |
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525766#comment-16525766
]
genericqa commented on HDFS-13485:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 32m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 30m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13485 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929485/HDFS-13485.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 82301f8120dc 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8752a48 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24512/testReport/ |
| Max. process+thread count | 1350 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24512/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL:
[jira] [Commented] (HDDS-195) Create generic CommandWatcher utility
[ https://issues.apache.org/jira/browse/HDDS-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525760#comment-16525760 ] Anu Engineer commented on HDDS-195: --- Looks good overall. Some minor comments. +1 after fixing them. * trackedEventsByUUID and trackedEvents can be inconsistent. if and only if, we have a hashCode for an object which is same.Say someone writes a hashcode as return 1; we can get into trouble. But, I think it is something that we should comment rather than solve. * Perhaps in a different JIRA, we create a leaseManager that can take the name as a parameter, so we know what these threads are doing when we debug ? * LeaseManager creates a thread pool per instance, should we have just one LeaseManager shared by all instances of Watchers ? * Spurious Edit in HddsDatanodeService.java > Create generic CommandWatcher utility > - > > Key: HDDS-195 > URL: https://issues.apache.org/jira/browse/HDDS-195 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-195.001.patch, HDDS-195.002.patch, > HDDS-195.003.patch > > > In some cases we need a class which can track the status of the outgoing > commands. > The commands should be resent after a while except a status message is > received about command completion. > On high level, we need a builder factory, which takes the following > parameters: > * (destination) event type and the payload of the command which should be > repeated. > * the ID of the command/event > * The event/topic of the completion messages. If an IdentifiableEventPayload > is received on this topic, the specific event is done and don't need to > resend it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-183) Integrate Volumeset, ContainerSet and HddsDispatcher
[ https://issues.apache.org/jira/browse/HDDS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525744#comment-16525744 ] Bharat Viswanadham edited comment on HDDS-183 at 6/27/18 11:56 PM: --- Hi [~hanishakoneru] Thanks for review. Addressed review comments in patch v04. ConstructKeyValueContainerData I have not separated it out, left in the same file. As this is Construct for KeyValueContainerData type, and it uses methods from Constructor. So for now, left as it is. was (Author: bharatviswa): Hi [~hanishakoneru] Thanks for review. Addressed review comments in patch v04. ConstructKeyValueContainerData I have not separated it out, left in the same file. As this is Construct for KeyValueContainerData type, and it uses methods from Constructor. So for now, left as it is. > Integrate Volumeset, ContainerSet and HddsDispatcher > > > Key: HDDS-183 > URL: https://issues.apache.org/jira/browse/HDDS-183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-183-HDDS-48.00.patch, HDDS-183-HDDS-48.01.patch, > HDDS-183-HDDS-48.02.patch, HDDS-183-HDDS-48.03.patch, > HDDS-183-HDDS-48.04.patch > > > This Jira adds following: > 1. Use new VolumeSet. > 2. build container map from .container files during startup. > 3. Integrate HddsDispatcher. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-183) Integrate Volumeset, ContainerSet and HddsDispatcher
[ https://issues.apache.org/jira/browse/HDDS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525744#comment-16525744 ] Bharat Viswanadham commented on HDDS-183: - Hi [~hanishakoneru] Thanks for review. Addressed review comments in patch v04. ConstructKeyValueContainerData I have not separated it out, left in the same file. As this is Construct for KeyValueContainerData type, and it uses methods from Constructor. So for now, left as it is. > Integrate Volumeset, ContainerSet and HddsDispatcher > > > Key: HDDS-183 > URL: https://issues.apache.org/jira/browse/HDDS-183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-183-HDDS-48.00.patch, HDDS-183-HDDS-48.01.patch, > HDDS-183-HDDS-48.02.patch, HDDS-183-HDDS-48.03.patch, > HDDS-183-HDDS-48.04.patch > > > This Jira adds following: > 1. Use new VolumeSet. > 2. build container map from .container files during startup. > 3. Integrate HddsDispatcher. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-183) Integrate Volumeset, ContainerSet and HddsDispatcher
[ https://issues.apache.org/jira/browse/HDDS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-183: Attachment: HDDS-183-HDDS-48.04.patch > Integrate Volumeset, ContainerSet and HddsDispatcher > > > Key: HDDS-183 > URL: https://issues.apache.org/jira/browse/HDDS-183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-183-HDDS-48.00.patch, HDDS-183-HDDS-48.01.patch, > HDDS-183-HDDS-48.02.patch, HDDS-183-HDDS-48.03.patch, > HDDS-183-HDDS-48.04.patch > > > This Jira adds following: > 1. Use new VolumeSet. > 2. build container map from .container files during startup. > 3. Integrate HddsDispatcher. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-173) Refactor Dispatcher and implement Handler for new ContainerIO design
[
https://issues.apache.org/jira/browse/HDDS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525729#comment-16525729
]
genericqa commented on HDDS-173:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} HDDS-48 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
29s{color} | {color:green} HDDS-48 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 23m
10s{color} | {color:red} root in HDDS-48 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} HDDS-48 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
52s{color} | {color:green} HDDS-48 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 5m
44s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} HDDS-48 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} HDDS-48 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 21m
57s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 21m 57s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 21m 57s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
20s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdds/container-service generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} tools in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 58s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[
https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525723#comment-16525723
]
genericqa commented on HDFS-13702:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 44s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
25s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13702 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929270/hdfs-13702.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fc9dd14165bc 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / aaf03cc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit
[
https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525720#comment-16525720
]
genericqa commented on HDFS-13703:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
32s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929463/hdfs-13703.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 23850b07e60b 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality
[jira] [Commented] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525707#comment-16525707
]
Siyao Meng commented on HDFS-13485:
---
Added sanity check for argument newValue in function decodeWritable().
I believe we don't need to check if obj is null since decodeWritable() is only
called from decodeFromUrlString() like this:
{code:java}
public void decodeFromUrlString(String newValue) throws IOException {
decodeWritable(this, newValue);
}
{code}
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL: https://issues.apache.org/jira/browse/HDFS-13485
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
>Affects Versions: 3.0.0
> Environment: Kerberized. Hadoop 3.0.0, WebHDFS.
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-13485.001.patch
>
>
> curl -k -i --negotiate -u : "https://hadoop3-4.example.com:20004/webhdfs/v1;
> DataNode Web UI should do a better error checking/handling.
> {noformat}
> 2018-04-19 10:07:49,338 WARN
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.token.Token.decodeWritable(Token.java:364)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:383)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.ParameterParser.delegationToken(ParameterParser.java:128)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.DataNodeUGIProvider.ugi(DataNodeUGIProvider.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:129)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:51)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:31)
> at
> com.cloudera.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1379)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
>
[jira] [Comment Edited] (HDFS-13524) Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
[
https://issues.apache.org/jira/browse/HDFS-13524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525689#comment-16525689
]
Siyao Meng edited comment on HDFS-13524 at 6/27/18 10:32 PM:
-
Increased the number of DataNodes from 1(default) to 3 when running the test.
The number of replicas is 1, indicated by createFile() argument #3.
was (Author: smeng):
Increased the number of DataNodes from 1(default) to 3 when running the test.
The number of replicas is 1, indicated by createFile() argument 3.
> Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
> -
>
> Key: HDFS-13524
> URL: https://issues.apache.org/jira/browse/HDFS-13524
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13524.001.patch
>
>
> TestLargeBlock#testLargeBlockSize may fail with error:
> {quote}
> All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:44968,DS-acddd79e-cdf1-4ac5-aac5-e804a2e61600,DISK]]
> are bad. Aborting...
> {quote}
> Tracing back, the error is due to the stress applied to the host sending a
> 2GB block, causing write pipeline ack read timeout:
> {quote}
> 2017-09-10 22:16:07,285 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (DataXceiver.java:writeBlock(742)) - Receiving
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001 src:
> /127.0.0.1:57794 dest: /127.0.0.1:44968
> 2017-09-10 22:16:50,402 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] WARN
> datanode.DataNode (BlockReceiver.java:flushOrSync(434)) - Slow flushOrSync
> took 5383ms (threshold=300ms), isSync:false, flushTotalNanos=5383638982ns,
> volume=file:/tmp/tmp.1oS3ZfDCwq/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/
> 2017-09-10 22:17:54,427 [ResponseProcessor for block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001] WARN
> hdfs.DataStreamer (DataStreamer.java:run(1214)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.net.SocketTimeoutException: 65000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/127.0.0.1:57794 remote=/127.0.0.1:44968]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:434)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
> at
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1104)
> 2017-09-10 22:17:54,432 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (BlockReceiver.java:receiveBlock(1000)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.io.IOException: Connection reset by peer
> {quote}
> Instead of raising read timeout, I suggest increasing cluster size from
> default=1 to 3, so that it has the opportunity to choose a different DN and
> retry.
> Suspect this fails after HDFS-13103, in Hadoop 2.8/3.0.0-alpha1 when we
> introduced client acknowledgement read timeout.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13524) Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
[
https://issues.apache.org/jira/browse/HDFS-13524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525689#comment-16525689
]
Siyao Meng edited comment on HDFS-13524 at 6/27/18 10:32 PM:
-
Increased the number of DataNodes from 1(default) to 3 when running the test.
The number of replicas is still 1, indicated by createFile() argument #3.
was (Author: smeng):
Increased the number of DataNodes from 1(default) to 3 when running the test.
The number of replicas is 1, indicated by createFile() argument #3.
> Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
> -
>
> Key: HDFS-13524
> URL: https://issues.apache.org/jira/browse/HDFS-13524
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13524.001.patch
>
>
> TestLargeBlock#testLargeBlockSize may fail with error:
> {quote}
> All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:44968,DS-acddd79e-cdf1-4ac5-aac5-e804a2e61600,DISK]]
> are bad. Aborting...
> {quote}
> Tracing back, the error is due to the stress applied to the host sending a
> 2GB block, causing write pipeline ack read timeout:
> {quote}
> 2017-09-10 22:16:07,285 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (DataXceiver.java:writeBlock(742)) - Receiving
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001 src:
> /127.0.0.1:57794 dest: /127.0.0.1:44968
> 2017-09-10 22:16:50,402 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] WARN
> datanode.DataNode (BlockReceiver.java:flushOrSync(434)) - Slow flushOrSync
> took 5383ms (threshold=300ms), isSync:false, flushTotalNanos=5383638982ns,
> volume=file:/tmp/tmp.1oS3ZfDCwq/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/
> 2017-09-10 22:17:54,427 [ResponseProcessor for block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001] WARN
> hdfs.DataStreamer (DataStreamer.java:run(1214)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.net.SocketTimeoutException: 65000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/127.0.0.1:57794 remote=/127.0.0.1:44968]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:434)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
> at
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1104)
> 2017-09-10 22:17:54,432 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (BlockReceiver.java:receiveBlock(1000)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.io.IOException: Connection reset by peer
> {quote}
> Instead of raising read timeout, I suggest increasing cluster size from
> default=1 to 3, so that it has the opportunity to choose a different DN and
> retry.
> Suspect this fails after HDFS-13103, in Hadoop 2.8/3.0.0-alpha1 when we
> introduced client acknowledgement read timeout.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13524) Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
[
https://issues.apache.org/jira/browse/HDFS-13524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-13524:
--
Attachment: HDFS-13524.001.patch
Status: Patch Available (was: In Progress)
Increased the number of DataNodes from 1(default) to 3 when running the test.
The number of replicas is 1, indicated by createFile() argument 3.
> Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
> -
>
> Key: HDFS-13524
> URL: https://issues.apache.org/jira/browse/HDFS-13524
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1, 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13524.001.patch
>
>
> TestLargeBlock#testLargeBlockSize may fail with error:
> {quote}
> All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:44968,DS-acddd79e-cdf1-4ac5-aac5-e804a2e61600,DISK]]
> are bad. Aborting...
> {quote}
> Tracing back, the error is due to the stress applied to the host sending a
> 2GB block, causing write pipeline ack read timeout:
> {quote}
> 2017-09-10 22:16:07,285 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (DataXceiver.java:writeBlock(742)) - Receiving
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001 src:
> /127.0.0.1:57794 dest: /127.0.0.1:44968
> 2017-09-10 22:16:50,402 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] WARN
> datanode.DataNode (BlockReceiver.java:flushOrSync(434)) - Slow flushOrSync
> took 5383ms (threshold=300ms), isSync:false, flushTotalNanos=5383638982ns,
> volume=file:/tmp/tmp.1oS3ZfDCwq/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/
> 2017-09-10 22:17:54,427 [ResponseProcessor for block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001] WARN
> hdfs.DataStreamer (DataStreamer.java:run(1214)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.net.SocketTimeoutException: 65000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/127.0.0.1:57794 remote=/127.0.0.1:44968]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:434)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
> at
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1104)
> 2017-09-10 22:17:54,432 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (BlockReceiver.java:receiveBlock(1000)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.io.IOException: Connection reset by peer
> {quote}
> Instead of raising read timeout, I suggest increasing cluster size from
> default=1 to 3, so that it has the opportunity to choose a different DN and
> retry.
> Suspect this fails after HDFS-13103, in Hadoop 2.8/3.0.0-alpha1 when we
> introduced client acknowledgement read timeout.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13524) Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
[
https://issues.apache.org/jira/browse/HDFS-13524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-13524 started by Siyao Meng.
-
> Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
> -
>
> Key: HDFS-13524
> URL: https://issues.apache.org/jira/browse/HDFS-13524
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Major
>
> TestLargeBlock#testLargeBlockSize may fail with error:
> {quote}
> All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:44968,DS-acddd79e-cdf1-4ac5-aac5-e804a2e61600,DISK]]
> are bad. Aborting...
> {quote}
> Tracing back, the error is due to the stress applied to the host sending a
> 2GB block, causing write pipeline ack read timeout:
> {quote}
> 2017-09-10 22:16:07,285 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (DataXceiver.java:writeBlock(742)) - Receiving
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001 src:
> /127.0.0.1:57794 dest: /127.0.0.1:44968
> 2017-09-10 22:16:50,402 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] WARN
> datanode.DataNode (BlockReceiver.java:flushOrSync(434)) - Slow flushOrSync
> took 5383ms (threshold=300ms), isSync:false, flushTotalNanos=5383638982ns,
> volume=file:/tmp/tmp.1oS3ZfDCwq/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/
> 2017-09-10 22:17:54,427 [ResponseProcessor for block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001] WARN
> hdfs.DataStreamer (DataStreamer.java:run(1214)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.net.SocketTimeoutException: 65000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/127.0.0.1:57794 remote=/127.0.0.1:44968]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:434)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
> at
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1104)
> 2017-09-10 22:17:54,432 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (BlockReceiver.java:receiveBlock(1000)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.io.IOException: Connection reset by peer
> {quote}
> Instead of raising read timeout, I suggest increasing cluster size from
> default=1 to 3, so that it has the opportunity to choose a different DN and
> retry.
> Suspect this fails after HDFS-13103, in Hadoop 2.8/3.0.0-alpha1 when we
> introduced client acknowledgement read timeout.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-193) Make Datanode heartbeat dispatcher in SCM event based
[ https://issues.apache.org/jira/browse/HDDS-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525677#comment-16525677 ] Hudson commented on HDDS-193: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14491 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14491/]) HDDS-193. Make Datanode heartbeat dispatcher in SCM event based. (aengineer: rev 8752a48564028cb5892c19e29d4e5b984d70c076) * (add) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMDatanodeHeartbeatDispatcher.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/report/SCMDatanodeReportHandler.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/report/TestSCMDatanodeNodeReportHandler.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/report/package-info.java * (delete) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestSCMMetrics.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/report/SCMDatanodeContainerReportHandler.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeHeartbeatDispatcher.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/report/package-info.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/report/SCMDatanodeReportHandlerFactory.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/report/TestSCMDatanodeContainerReportHandler.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/report/TestSCMDatanodeReportHandlerFactory.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/report/SCMDatanodeHeartbeatDispatcher.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/report/SCMDatanodeNodeReportHandler.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/report/TestSCMDatanodeHeartbeatDispatcher.java > Make Datanode heartbeat dispatcher in SCM event based > - > > Key: HDDS-193 > URL: https://issues.apache.org/jira/browse/HDDS-193 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-193.001.patch, HDDS-193.002.patch, > HDDS-193.003.patch, HDDS-193.004.patch, HDDS-193.004.patch > > > HDDS-163 introduced a new dispatcher in the SCM side to send the heartbeat > report parts to the appropriate listeners. I propose to make it EventQueue > based to handle/monitor these async calls in the same way as the other events. > Report handlers would subscribe to the specific events to process the > information. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-94) Change ozone datanode command to start the standalone datanode plugin
[ https://issues.apache.org/jira/browse/HDDS-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525660#comment-16525660 ] Hudson commented on HDDS-94: SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14490 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14490/]) HDDS-94. Change ozone datanode command to start the standalone datanode (aengineer: rev 18932717c42382ed8842de7719ec6d20c1765366) * (edit) hadoop-ozone/acceptance-test/src/test/acceptance/basic/docker-compose.yaml * (edit) hadoop-ozone/common/src/main/bin/ozone * (edit) hadoop-ozone/acceptance-test/src/test/acceptance/basic/docker-config * (edit) hadoop-ozone/acceptance-test/src/test/acceptance/ozonefs/docker-config * (edit) hadoop-dist/src/main/compose/ozoneperf/docker-compose.yaml * (edit) hadoop-ozone/acceptance-test/src/test/acceptance/ozonefs/docker-compose.yaml * (edit) hadoop-dist/src/main/compose/ozone/docker-compose.yaml * (edit) hadoop-ozone/acceptance-test/src/test/acceptance/commonlib.robot * (edit) hadoop-dist/src/main/compose/ozoneperf/docker-config * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java * (edit) hadoop-dist/src/main/compose/ozone/docker-config > Change ozone datanode command to start the standalone datanode plugin > - > > Key: HDDS-94 > URL: https://issues.apache.org/jira/browse/HDDS-94 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Sandeep Nemuri >Priority: Major > Labels: newbie > Fix For: 0.2.1 > > Attachments: HDDS-94.001.patch, HDDS-94.002.patch, HDDS-94.003.patch, > HDDS-94.004.patch, HDDS-94.005.patch > > > The current ozone datanode command starts the regular hdfs datanode with an > enabled HddsDatanodeService as a datanode plugin. > The goal is to start only the HddsDatanodeService.java (main function is > already there but GenericOptionParser should be adopted). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-186) Create under replicated queue
[ https://issues.apache.org/jira/browse/HDDS-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525658#comment-16525658 ] Ajay Kumar commented on HDDS-186: - [~xyao] thanks for review and commit. > Create under replicated queue > - > > Key: HDDS-186 > URL: https://issues.apache.org/jira/browse/HDDS-186 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-186.00.patch, HDDS-186.01.patch, HDDS-186.02.patch, > HDDS-186.03.patch > > > Create under replicated queue to replicate under replicated containers in > Ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-13485:
--
Fix Version/s: 3.0.0
Attachment: HDFS-13485.001.patch
Status: Patch Available (was: In Progress)
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL: https://issues.apache.org/jira/browse/HDFS-13485
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
>Affects Versions: 3.0.0
> Environment: Kerberized. Hadoop 3.0.0, WebHDFS.
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-13485.001.patch
>
>
> curl -k -i --negotiate -u : "https://hadoop3-4.example.com:20004/webhdfs/v1;
> DataNode Web UI should do a better error checking/handling.
> {noformat}
> 2018-04-19 10:07:49,338 WARN
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.token.Token.decodeWritable(Token.java:364)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:383)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.ParameterParser.delegationToken(ParameterParser.java:128)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.DataNodeUGIProvider.ugi(DataNodeUGIProvider.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:129)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:51)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:31)
> at
> com.cloudera.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1379)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
>
[jira] [Commented] (HDDS-167) Rename KeySpaceManager to OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525656#comment-16525656
]
genericqa commented on HDDS-167:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDDS-167 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDDS-167 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929217/HDDS-167.05.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/377/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Rename KeySpaceManager to OzoneManager
> --
>
> Key: HDDS-167
> URL: https://issues.apache.org/jira/browse/HDDS-167
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: Ozone Manager
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-167.01.patch, HDDS-167.02.patch, HDDS-167.03.patch,
> HDDS-167.04.patch, HDDS-167.05.patch
>
>
> The Ozone KeySpaceManager daemon was renamed to OzoneManager. There's some
> more changes needed to complete the rename everywhere e.g.
> - command-line
> - documentation
> - unit tests
> - Acceptance tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-193) Make Datanode heartbeat dispatcher in SCM event based
[ https://issues.apache.org/jira/browse/HDDS-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-193: -- Resolution: Fixed Status: Resolved (was: Patch Available) [~elek] Thanks for taking care of this issue. I have committed this to trunk > Make Datanode heartbeat dispatcher in SCM event based > - > > Key: HDDS-193 > URL: https://issues.apache.org/jira/browse/HDDS-193 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-193.001.patch, HDDS-193.002.patch, > HDDS-193.003.patch, HDDS-193.004.patch, HDDS-193.004.patch > > > HDDS-163 introduced a new dispatcher in the SCM side to send the heartbeat > report parts to the appropriate listeners. I propose to make it EventQueue > based to handle/monitor these async calls in the same way as the other events. > Report handlers would subscribe to the specific events to process the > information. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-170) Fix TestBlockDeletingService#testBlockDeletionTimeout
[ https://issues.apache.org/jira/browse/HDDS-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525640#comment-16525640 ] Hudson commented on HDDS-170: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14489 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14489/]) HDDS-170. Fix TestBlockDeletingService#testBlockDeletionTimeout. (xyao: rev 1e30547642c7c6c014745862dd06f90f091f90b6) * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/BackgroundService.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java * (edit) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/testutils/BlockDeletingServiceTestImpl.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/background/BlockDeletingService.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java > Fix TestBlockDeletingService#testBlockDeletionTimeout > - > > Key: HDDS-170 > URL: https://issues.apache.org/jira/browse/HDDS-170 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-170.001.patch > > > TestBlockDeletingService#testBlockDeletionTimeout timesout while waiting for > expected error messsage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-186) Create under replicated queue
[ https://issues.apache.org/jira/browse/HDDS-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525639#comment-16525639 ] Hudson commented on HDDS-186: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14489 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14489/]) HDDS-186. Create under replicated queue. Contributed by Ajay Kumar. (xyao: rev e9ec3d78f520a8543dc77d763d4b358aa608bae8) * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationRequest.java * (add) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationQueue.java * (add) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/package-info.java * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/package-info.java * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationQueue.java > Create under replicated queue > - > > Key: HDDS-186 > URL: https://issues.apache.org/jira/browse/HDDS-186 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-186.00.patch, HDDS-186.01.patch, HDDS-186.02.patch, > HDDS-186.03.patch > > > Create under replicated queue to replicate under replicated containers in > Ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-173) Refactor Dispatcher and implement Handler for new ContainerIO design
[ https://issues.apache.org/jira/browse/HDDS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525630#comment-16525630 ] Hanisha Koneru commented on HDDS-173: - Fixed Junit test failures from last run in patch v05. > Refactor Dispatcher and implement Handler for new ContainerIO design > > > Key: HDDS-173 > URL: https://issues.apache.org/jira/browse/HDDS-173 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-173-HDDS-48.001.patch, HDDS-173-HDDS-48.002.patch, > HDDS-173-HDDS-48.003.patch, HDDS-173-HDDS-48.004.patch, > HDDS-173-HDDS-48.005.patch > > > Dispatcher will pass the ContainerCommandRequests to the corresponding > Handler based on the ContainerType. Each ContainerType will have its own > Handler. The Handler class will process the message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-173) Refactor Dispatcher and implement Handler for new ContainerIO design
[ https://issues.apache.org/jira/browse/HDDS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-173: Attachment: HDDS-173-HDDS-48.005.patch > Refactor Dispatcher and implement Handler for new ContainerIO design > > > Key: HDDS-173 > URL: https://issues.apache.org/jira/browse/HDDS-173 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-173-HDDS-48.001.patch, HDDS-173-HDDS-48.002.patch, > HDDS-173-HDDS-48.003.patch, HDDS-173-HDDS-48.004.patch, > HDDS-173-HDDS-48.005.patch > > > Dispatcher will pass the ContainerCommandRequests to the corresponding > Handler based on the ContainerType. Each ContainerType will have its own > Handler. The Handler class will process the message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-94) Change ozone datanode command to start the standalone datanode plugin
[ https://issues.apache.org/jira/browse/HDDS-94?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-94: - Resolution: Fixed Status: Resolved (was: Patch Available) [~elek] Thanks for review and additions. [~Sandeep Nemuri] Thanks for the contribution. > Change ozone datanode command to start the standalone datanode plugin > - > > Key: HDDS-94 > URL: https://issues.apache.org/jira/browse/HDDS-94 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Sandeep Nemuri >Priority: Major > Labels: newbie > Fix For: 0.2.1 > > Attachments: HDDS-94.001.patch, HDDS-94.002.patch, HDDS-94.003.patch, > HDDS-94.004.patch, HDDS-94.005.patch > > > The current ozone datanode command starts the regular hdfs datanode with an > enabled HddsDatanodeService as a datanode plugin. > The goal is to start only the HddsDatanodeService.java (main function is > already there but GenericOptionParser should be adopted). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525612#comment-16525612 ] stack commented on HDFS-13702: -- bq. I do want a trace layer in there; I do want it broader than just HDFS, and I do want it to be used from the layers above. Me too. bq. Otherwise: it'll get cut, nobody will replace it, and it'll get lost in folklore. Better this than a dead, disabled lib burning everyone's CPU to no end. Even when enabled, as is, it is of little to no value. Trace in hdfs is in need of work but it has been suffering neglect since Colin's add. bq. This is something to talk about at a broader level than a JIRA; I can start a thread. Suggest this discussion not block this patch? Or, add in placeholders/comments for the trace points removed here? Thanks [~ste...@apache.org] > HTrace hooks taking 10-15% CPU in DFS client when disabled > -- > > Key: HDFS-13702 > URL: https://issues.apache.org/jira/browse/HDFS-13702 > Project: Hadoop HDFS > Issue Type: Bug > Components: performance >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch > > > I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate > workload even when HTrace is disabled. This is because it stringifies several > integers. We should avoid all allocation and stringification when htrace is > disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-194) Remove NodePoolManager and node pool handling from SCM
[ https://issues.apache.org/jira/browse/HDDS-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525611#comment-16525611 ] Hudson commented on HDDS-194: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14488 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14488/]) Revert "HDDS-194. Remove NodePoolManager and node pool handling from (xyao: rev 0d6fe5f36be5b19aab89d995866e526c5feec758) * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodePoolManager.java * (delete) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/package-info.java * (delete) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/package-info.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/MockNodeManager.java * (edit) hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/scm/cli/SQLCLI.java * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/InProgressPool.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerMapping.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java * (delete) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationQueue.java * (delete) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationReqMsg.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodePoolManager.java * (add) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/testutils/ReplicationNodePoolManagerMock.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/PeriodicPool.java * (add) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodePoolManager.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/package-info.java * (delete) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationQueue.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ContainerSupervisor.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/testutils/ReplicationNodeManagerMock.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSQLCli.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java HDDS-194. Remove NodePoolManager and node pool handling from SCM. (xyao: rev 56a4cdb9804daea7164155a5b1b4eba44a11b705) * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/testutils/ReplicationNodePoolManagerMock.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodePoolManager.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/MockNodeManager.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/testutils/ReplicationNodeManagerMock.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/InProgressPool.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/package-info.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodePoolManager.java * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerMapping.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodePoolManager.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSQLCli.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ContainerSupervisor.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/PeriodicPool.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (edit) hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/scm/cli/SQLCLI.java > Remove NodePoolManager and node pool handling from
[jira] [Comment Edited] (HDDS-170) Fix TestBlockDeletingService#testBlockDeletionTimeout
[ https://issues.apache.org/jira/browse/HDDS-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525601#comment-16525601 ] Xiaoyu Yao edited comment on HDDS-170 at 6/27/18 9:02 PM: -- Thanks [~ljain] for fixing this. I've tested it locally with/without the patch and verify the fix. +1, I will commit it shortly. was (Author: xyao): Thanks [~ljain] for fixing this. I've tested it locally with/without the patch and verify the fix. I will commit it shortly. > Fix TestBlockDeletingService#testBlockDeletionTimeout > - > > Key: HDDS-170 > URL: https://issues.apache.org/jira/browse/HDDS-170 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-170.001.patch > > > TestBlockDeletingService#testBlockDeletionTimeout timesout while waiting for > expected error messsage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-170) Fix TestBlockDeletingService#testBlockDeletionTimeout
[ https://issues.apache.org/jira/browse/HDDS-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-170: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~ljain] for the contribution. I've commit the patch to trunk. > Fix TestBlockDeletingService#testBlockDeletionTimeout > - > > Key: HDDS-170 > URL: https://issues.apache.org/jira/browse/HDDS-170 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-170.001.patch > > > TestBlockDeletingService#testBlockDeletionTimeout timesout while waiting for > expected error messsage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-170) Fix TestBlockDeletingService#testBlockDeletionTimeout
[ https://issues.apache.org/jira/browse/HDDS-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525601#comment-16525601 ] Xiaoyu Yao edited comment on HDDS-170 at 6/27/18 8:55 PM: -- Thanks [~ljain] for fixing this. I've tested it locally with/without the patch and verify the fix. I will commit it shortly. was (Author: xyao): Thanks [~ljain] for fixing this. I've tested it locally with/without the patch and verify the fix. This is a minor issue in TestBlockDeletingService.java line 283 where the comment is not updated. Instead of saying timeout 1ms, Timeout should be 1ns after this change. > Fix TestBlockDeletingService#testBlockDeletionTimeout > - > > Key: HDDS-170 > URL: https://issues.apache.org/jira/browse/HDDS-170 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-170.001.patch > > > TestBlockDeletingService#testBlockDeletionTimeout timesout while waiting for > expected error messsage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-170) Fix TestBlockDeletingService#testBlockDeletionTimeout
[ https://issues.apache.org/jira/browse/HDDS-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525601#comment-16525601 ] Xiaoyu Yao edited comment on HDDS-170 at 6/27/18 8:54 PM: -- Thanks [~ljain] for fixing this. I've tested it locally with/without the patch and verify the fix. This is a minor issue in TestBlockDeletingService.java line 283 where the comment is not updated. Instead of saying timeout 1ms, Timeout should be 1ns after this change. was (Author: xyao): Thanks [~ljain] for fixing this. I've tested it locally with/without the patch and verify the fix. This is a minor issue in TestBlockDeletingService.java line 283 where the comment is not updated. Instead of saying 1ms, it should be 1ns after this change. > Fix TestBlockDeletingService#testBlockDeletionTimeout > - > > Key: HDDS-170 > URL: https://issues.apache.org/jira/browse/HDDS-170 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-170.001.patch > > > TestBlockDeletingService#testBlockDeletionTimeout timesout while waiting for > expected error messsage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-170) Fix TestBlockDeletingService#testBlockDeletionTimeout
[ https://issues.apache.org/jira/browse/HDDS-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525601#comment-16525601 ] Xiaoyu Yao commented on HDDS-170: - Thanks [~ljain] for fixing this. I've tested it locally with/without the patch and verify the fix. This is a minor issue in TestBlockDeletingService.java line 283 where the comment is not updated. Instead of saying 1ms, it should be 1ns after this change. > Fix TestBlockDeletingService#testBlockDeletionTimeout > - > > Key: HDDS-170 > URL: https://issues.apache.org/jira/browse/HDDS-170 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-170.001.patch > > > TestBlockDeletingService#testBlockDeletionTimeout timesout while waiting for > expected error messsage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-186) Create under replicated queue
[ https://issues.apache.org/jira/browse/HDDS-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-186: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~ajayydv] for the contribution. I've commit the patch to trunk. > Create under replicated queue > - > > Key: HDDS-186 > URL: https://issues.apache.org/jira/browse/HDDS-186 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-186.00.patch, HDDS-186.01.patch, HDDS-186.02.patch, > HDDS-186.03.patch > > > Create under replicated queue to replicate under replicated containers in > Ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-183) Integrate Volumeset, ContainerSet and HddsDispatcher
[
https://issues.apache.org/jira/browse/HDDS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525589#comment-16525589
]
Hanisha Koneru edited comment on HDDS-183 at 6/27/18 8:40 PM:
--
Thanks for working on this [~bharatviswa].
A few comments:
* ContainerReader
** Line 106, 112 : if check redundant
** Line 160, 164 : Catch blocks can be combined as one.
** Line 147 : We should not cast to KeyValueContainerData
** verifyContainerFile() - If containerType does not match KeyValueContainer,
we should log an error.
** parseKeyValueContainerData() can have void return type as the containerData
is updated in place.
Also, can we move this function to KeyValueContainerUtils or some other
KeyValue class. It would be good to keep the common implementations free of
specific containerType code.
** ContainerFilter class is unused.
* ContainerDataYaml
** Line 84, 174 - new Tag("KeyValueContainerData”) should be defined as a
static variable in KeyValueContainerData or KeyValueContainer class.
** Line 207 - state can be “INVALID” also.
** Line 152-157 : Instead of defining the valid fields here, can we maintain a
list of valid yaml fields for KVContainer in KVContainerData. This way, if a
new field is added to KVContainerData, we can update this list itself and avoid
"When a new field needs to be added, it needs to be added here.” Also, these
fields should be defined as static final varaibles.
** It would be good to move ConstructKeyValueContainerData class also to
keyvalue package.
* DeleteBlocksCommandHandler - we should handle different containerTypes here.
For types other than KV, we can just log an error.
* VersionEndpointTask
** Line 70 : keyValues not used.
** Line 84 : ozoneContainer.getDispatcher().setScmId(scmId); should be outside
the for loop for volumeMap.
* HddsVolume - changes are not required.
* VolumeSet - NIT - Line 358 : Unused variable srb
* KeyValueContainerUtil - NIT - Line 213 : check sum -> checksum, Container
Name -> Container ID (just to avoid confusion for admins when debugging logs).
* KeyValueHandler
** We need the kvContainer.writeUnlock() in handleDeleteContainer(). When we
detect that the container is open, we should immediately release the lock and
not wait for any other operation (even loggging).
* SCMTestUtils - can we replace DFS_DATANODE_DATA_DIR_KEY with
HDDS_DATANODE_DIR_KEY.
was (Author: hanishakoneru):
Thanks for working on this [~bharatviswa].
A few comments:
* ContainerReader
** Line 106, 112 : if check redundant
** Line 160, 164 : Catch blocks can be combined as one.
** Line 147 : We should not cast to KeyValueContainerData
** verifyContainerFile() - If containerType does not match KeyValueContainer,
we should log an error.
** parseKeyValueContainerData() can have void return type as the containerData
is updated in place.
Also, can we move this function to KeyValueContainerUtils or some other
KeyValue class. It would be good to keep the common implementations free of
specific containerType code.
** ContainerFilter class is unused.
* ContainerDataYaml
** Line 84, 174 - new Tag("KeyValueContainerData”) should be defined as a
static variable in KeyValueContainerData or KeyValueContainer class.
** Line 207 - state can be “INVALID” also.
** Line 152-157 : Instead of defining the valid fields here, can we maintain a
list of valid yaml fields for KVContainer in KVContainerData. This way, if a
new field is added to KVContainerData, we can update this list itself and avoid
"When a new field needs to be added, it needs to be added here.” Also, these
fields should be defined as static final varaibles.
** It would be good to move ConstructKeyValueContainerData class also to
keyvalue package.
* DeleteBlocksCommandHandler - we should handle different containerTypes here.
For types other than KV, we can just log an error.
* VersionEndpointTask
** Line 70 : keyValues not used.
** Line 84 : ozoneContainer.getDispatcher().setScmId(scmId); should be outside
the for loop for volumeMap.
* HddsVolume - changes are not required.
* VolumeSet - NIT - Line 358 : Unused variable srb
* KeyValueContainerUtil - NIT - Line 213 : check sum -> checksum, Container
Name -> Container ID (just to avoid confusion for admins when debugging logs).
* KeyValueHandler
** We need the kvContainer.writeUnlock() in handleDeleteContainer(). When we
detect that the container is open, we should immediately release the lock and
not wait for any other operation (even loggging).
> Integrate Volumeset, ContainerSet and HddsDispatcher
>
>
> Key: HDDS-183
> URL: https://issues.apache.org/jira/browse/HDDS-183
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
[jira] [Commented] (HDDS-183) Integrate Volumeset, ContainerSet and HddsDispatcher
[
https://issues.apache.org/jira/browse/HDDS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525589#comment-16525589
]
Hanisha Koneru commented on HDDS-183:
-
Thanks for working on this [~bharatviswa].
A few comments:
* ContainerReader
** Line 106, 112 : if check redundant
** Line 160, 164 : Catch blocks can be combined as one.
** Line 147 : We should not cast to KeyValueContainerData
** verifyContainerFile() - If containerType does not match KeyValueContainer,
we should log an error.
** parseKeyValueContainerData() can have void return type as the containerData
is updated in place.
Also, can we move this function to KeyValueContainerUtils or some other
KeyValue class. It would be good to keep the common implementations free of
specific containerType code.
** ContainerFilter class is unused.
* ContainerDataYaml
** Line 84, 174 - new Tag("KeyValueContainerData”) should be defined as a
static variable in KeyValueContainerData or KeyValueContainer class.
** Line 207 - state can be “INVALID” also.
** Line 152-157 : Instead of defining the valid fields here, can we maintain a
list of valid yaml fields for KVContainer in KVContainerData. This way, if a
new field is added to KVContainerData, we can update this list itself and avoid
"When a new field needs to be added, it needs to be added here.” Also, these
fields should be defined as static final varaibles.
** It would be good to move ConstructKeyValueContainerData class also to
keyvalue package.
* DeleteBlocksCommandHandler - we should handle different containerTypes here.
For types other than KV, we can just log an error.
* VersionEndpointTask
** Line 70 : keyValues not used.
** Line 84 : ozoneContainer.getDispatcher().setScmId(scmId); should be outside
the for loop for volumeMap.
* HddsVolume - changes are not required.
* VolumeSet - NIT - Line 358 : Unused variable srb
* KeyValueContainerUtil - NIT - Line 213 : check sum -> checksum, Container
Name -> Container ID (just to avoid confusion for admins when debugging logs).
* KeyValueHandler
** We need the kvContainer.writeUnlock() in handleDeleteContainer(). When we
detect that the container is open, we should immediately release the lock and
not wait for any other operation (even loggging).
> Integrate Volumeset, ContainerSet and HddsDispatcher
>
>
> Key: HDDS-183
> URL: https://issues.apache.org/jira/browse/HDDS-183
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-183-HDDS-48.00.patch, HDDS-183-HDDS-48.01.patch,
> HDDS-183-HDDS-48.02.patch, HDDS-183-HDDS-48.03.patch
>
>
> This Jira adds following:
> 1. Use new VolumeSet.
> 2. build container map from .container files during startup.
> 3. Integrate HddsDispatcher.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-194) Remove NodePoolManager and node pool handling from SCM
[ https://issues.apache.org/jira/browse/HDDS-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525591#comment-16525591 ] Xiaoyu Yao commented on HDDS-194: - The previous commit accidentally included some of the changes from HDDS-186. I've reverted and recommit it. Sorry for the confusion. > Remove NodePoolManager and node pool handling from SCM > -- > > Key: HDDS-194 > URL: https://issues.apache.org/jira/browse/HDDS-194 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-194.001.patch, HDDS-194.002.patch > > > The current code use NodePoolManager and ContainerSupervisor to group the > nodes to smaller groups (pools) and handle the pull based node reports group > by group. > But this code is not used any more as we switch back to use a push based > model. In the datanode the reports could be handled by the specific report > handlers, and in the scm side the reports will be processed by the > SCMHeartbeatDispatcher which will send the events to the EventQueue. > As of now the NodePool abstraction could be removed from the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12976) Introduce ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525574#comment-16525574
]
Chao Sun edited comment on HDFS-12976 at 6/27/18 8:28 PM:
--
Thanks for the comments [~xkrogen] and [~shv]!
[~xkrogen]: yes we can potentially use {{stateId}} to achieve the same purpose
- totally agree with you on the concern about adding another flag which
potentially is only used for talking to NameNode.
[~shv]: I like the idea of {{T extends HAServiceProtocol}}, however it may need
more change as {{NameNodeProxiesClient#createProxyWithClientProtocol}} only
create the {{T}} with {{ClientProtocol}}, which may cause exception like:
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB cannot be
cast to org.apache.hadoop.ha.HAServiceProtocol
{code}
Sorry I totally forgot about the alignment changes. Thanks for doing that.
Attached the v004 patch after rebasing to the latest HDFS-12943.
was (Author: csun):
Thanks for the comments [~xkrogen] and [~shv]!
[~xkrogen]: yes we can potentially use {{stateId}} to achieve the same purpose
- totally agree with you on the concern about adding another flag which
potentially is only used for talking to NameNode.
[~shv]: I like the idea of {{T extends HAServiceProtocol}}, however it may need
more change as {{NameNodeProxiesClient#createProxyWithClientProtocol}} only
create the {{T}} with {{ClientProtocol}}. Otherwise you may get exception like:
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB cannot be
cast to org.apache.hadoop.ha.HAServiceProtocol
{code}
Sorry I totally forgot about the alignment changes. Thanks for doing that.
Attached the v004 patch after rebasing to the latest HDFS-12943.
> Introduce ObserverReadProxyProvider
> ---
>
> Key: HDFS-12976
> URL: https://issues.apache.org/jira/browse/HDFS-12976
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12976-HDFS-12943.000.patch,
> HDFS-12976-HDFS-12943.001.patch, HDFS-12976-HDFS-12943.002.patch,
> HDFS-12976-HDFS-12943.003.patch, HDFS-12976-HDFS-12943.004.patch,
> HDFS-12976-HDFS-12943.005.patch, HDFS-12976.WIP.patch
>
>
> {{StandbyReadProxyProvider}} should implement {{FailoverProxyProvider}}
> interface and be able to submit read requests to ANN and SBN(s).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12976) Introduce ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525574#comment-16525574
]
Chao Sun commented on HDFS-12976:
-
Thanks for the comments [~xkrogen] and [~shv]!
[~xkrogen]: yes we can potentially use {{stateId}} to achieve the same purpose
- totally agree with you on the concern about adding another flag which
potentially is only used for talking to NameNode.
[~shv]: I like the idea of {{T extends HAServiceProtocol}}, however it may need
more change as {{NameNodeProxiesClient#createProxyWithClientProtocol}} only
create the {{T}} with {{ClientProtocol}}. Otherwise you may get exception like:
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB cannot be
cast to org.apache.hadoop.ha.HAServiceProtocol
{code}
Sorry I totally forgot about the alignment changes. Thanks for doing that.
Attached the v004 patch after rebasing to the latest HDFS-12943.
> Introduce ObserverReadProxyProvider
> ---
>
> Key: HDFS-12976
> URL: https://issues.apache.org/jira/browse/HDFS-12976
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12976-HDFS-12943.000.patch,
> HDFS-12976-HDFS-12943.001.patch, HDFS-12976-HDFS-12943.002.patch,
> HDFS-12976-HDFS-12943.003.patch, HDFS-12976-HDFS-12943.004.patch,
> HDFS-12976-HDFS-12943.005.patch, HDFS-12976.WIP.patch
>
>
> {{StandbyReadProxyProvider}} should implement {{FailoverProxyProvider}}
> interface and be able to submit read requests to ANN and SBN(s).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[
https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525569#comment-16525569
]
Todd Lipcon commented on HDFS-13702:
{quote}I do want a trace layer in there; I do want it broader than just HDFS,
and I do want it to be used from the layers above. I don't want stuff taken
from DFSClient until there's a better story there. Otherwise: it'll get cut,
nobody will replace it, and it'll get lost in folklore.
{quote}
Sure, I agree tracing is nice. I recently worked on adding OpenTracing support
to the HMS (HIVE-19685). I think we could do the same here. That said,
regardless of trace framework, I think we can all agree that the current code
path is too expensive as written today. I don't have time to do the heavy
lifting of moving entirely to some new tracing framework, so in the meantime I
think we should fix this perf issue by removing this heavy code path. I didn't
remove all the other FS-operation-level trace annotations (eg
open/close/listStatus/etc) since those are likely to be less high throughput
and therefore overhead not an issue.
> HTrace hooks taking 10-15% CPU in DFS client when disabled
> --
>
> Key: HDFS-13702
> URL: https://issues.apache.org/jira/browse/HDFS-13702
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: performance
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Major
> Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch
>
>
> I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate
> workload even when HTrace is disabled. This is because it stringifies several
> integers. We should avoid all allocation and stringification when htrace is
> disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-186) Create under replicated queue
[ https://issues.apache.org/jira/browse/HDDS-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525564#comment-16525564 ] Xiaoyu Yao commented on HDDS-186: - +1 for v3 patch. I will commit it shortly. > Create under replicated queue > - > > Key: HDDS-186 > URL: https://issues.apache.org/jira/browse/HDDS-186 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-186.00.patch, HDDS-186.01.patch, HDDS-186.02.patch, > HDDS-186.03.patch > > > Create under replicated queue to replicate under replicated containers in > Ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12976) Introduce ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HDFS-12976:
Attachment: HDFS-12976-HDFS-12943.005.patch
> Introduce ObserverReadProxyProvider
> ---
>
> Key: HDFS-12976
> URL: https://issues.apache.org/jira/browse/HDFS-12976
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12976-HDFS-12943.000.patch,
> HDFS-12976-HDFS-12943.001.patch, HDFS-12976-HDFS-12943.002.patch,
> HDFS-12976-HDFS-12943.003.patch, HDFS-12976-HDFS-12943.004.patch,
> HDFS-12976-HDFS-12943.005.patch, HDFS-12976.WIP.patch
>
>
> {{StandbyReadProxyProvider}} should implement {{FailoverProxyProvider}}
> interface and be able to submit read requests to ANN and SBN(s).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525540#comment-16525540 ] Steve Loughran commented on HDFS-13702: --- This is something to talk about at a broader level than a JIRA; Stack said > I think we should commit this patch, +1, =0 I do want a trace layer in there; I do want it broader than just HDFS, and I do want it to be used from the layers above. I don't want stuff taken from DFSClient until there's a better story there. Otherwise: it'll get cut, nobody will replace it, and it'll get lost in folklore. > HTrace hooks taking 10-15% CPU in DFS client when disabled > -- > > Key: HDFS-13702 > URL: https://issues.apache.org/jira/browse/HDFS-13702 > Project: Hadoop HDFS > Issue Type: Bug > Components: performance >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch > > > I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate > workload even when HTrace is disabled. This is because it stringifies several > integers. We should avoid all allocation and stringification when htrace is > disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525536#comment-16525536 ] Todd Lipcon commented on HDFS-13702: Perhaps using '-nsu' on all the patch builds? Would be nice if there were a way to get the equivalent of 'nsu' but specifying a particular glob of artifact IDs, etc. > HTrace hooks taking 10-15% CPU in DFS client when disabled > -- > > Key: HDFS-13702 > URL: https://issues.apache.org/jira/browse/HDFS-13702 > Project: Hadoop HDFS > Issue Type: Bug > Components: performance >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch > > > I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate > workload even when HTrace is disabled. This is because it stringifies several > integers. We should avoid all allocation and stringification when htrace is > disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-194) Remove NodePoolManager and node pool handling from SCM
[ https://issues.apache.org/jira/browse/HDDS-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525532#comment-16525532 ] Hudson commented on HDDS-194: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14487 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14487/]) HDDS-194. Remove NodePoolManager and node pool handling from SCM. (xyao: rev aaf03cc459a34af284f9735453aefd4ddb430d67) * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodePoolManager.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java * (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml * (add) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/package-info.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/package-info.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationReqMsg.java * (add) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationQueue.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/PeriodicPool.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSQLCli.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/testutils/ReplicationNodeManagerMock.java * (edit) hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/scm/cli/SQLCLI.java * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationQueue.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ContainerSupervisor.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/InProgressPool.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/package-info.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/testutils/ReplicationNodePoolManagerMock.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodePoolManager.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerMapping.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/MockNodeManager.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodePoolManager.java > Remove NodePoolManager and node pool handling from SCM > -- > > Key: HDDS-194 > URL: https://issues.apache.org/jira/browse/HDDS-194 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-194.001.patch, HDDS-194.002.patch > > > The current code use NodePoolManager and ContainerSupervisor to group the > nodes to smaller groups (pools) and handle the pull based node reports group > by group. > But this code is not used any more as we switch back to use a push based > model. In the datanode the reports could be handled by the specific report > handlers, and in the scm side the reports will be processed by the > SCMHeartbeatDispatcher which will send the events to the EventQueue. > As of now the NodePool abstraction could be removed from the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-173) Refactor Dispatcher and implement Handler for new ContainerIO design
[
https://issues.apache.org/jira/browse/HDDS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525533#comment-16525533
]
genericqa commented on HDDS-173:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} HDDS-48 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
1s{color} | {color:green} HDDS-48 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 22m
38s{color} | {color:red} root in HDDS-48 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} HDDS-48 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} HDDS-48 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 5m
48s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} HDDS-48 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} HDDS-48 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 22m
26s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 22m 26s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 22m 26s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
13s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
54s{color} | {color:red} hadoop-hdds/container-service generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
7s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 59s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} tools in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 44s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color}
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[
https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525528#comment-16525528
]
Sean Busbey commented on HDFS-13702:
{quote}
Then, when it ran the hadoop-hdfs tests, they ran against the trunk snapshot
build rather than the patched snapshot build, and failed to compile. Sean's
going to file a YETUS JIRA about this and re-submit the patch build here.
{quote}
I wanna take a look at Hadoop's pom. ideally we shouldn't have the ASF snapshot
repo as a source of artifacts at all. I'm not sure there's a way for yetus to
proactively defend against it. (maybe use maven's {{--offline}} for build steps
after the one where we say we're downloading the world?)
> HTrace hooks taking 10-15% CPU in DFS client when disabled
> --
>
> Key: HDFS-13702
> URL: https://issues.apache.org/jira/browse/HDFS-13702
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: performance
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Major
> Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch
>
>
> I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate
> workload even when HTrace is disabled. This is because it stringifies several
> integers. We should avoid all allocation and stringification when htrace is
> disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525525#comment-16525525 ] Sean Busbey commented on HDFS-13702: well I started another run but the Hadoop related precommit jobs don't have a debug flag so I'll need to add one. > HTrace hooks taking 10-15% CPU in DFS client when disabled > -- > > Key: HDFS-13702 > URL: https://issues.apache.org/jira/browse/HDFS-13702 > Project: Hadoop HDFS > Issue Type: Bug > Components: performance >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch > > > I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate > workload even when HTrace is disabled. This is because it stringifies several > integers. We should avoid all allocation and stringification when htrace is > disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525524#comment-16525524 ] Todd Lipcon commented on HDFS-13702: Looked into this a bit more and found the issue: the build happened to run around midnight UTC, so the "patch javadoc" build downloaded a new SNAPSHOT build of the 'hadoop-hdfs-client' jar into the local repo. Then, when it ran the hadoop-hdfs tests, they ran against the trunk snapshot build rather than the patched snapshot build, and failed to compile. Sean's going to file a YETUS JIRA about this and re-submit the patch build here. > HTrace hooks taking 10-15% CPU in DFS client when disabled > -- > > Key: HDFS-13702 > URL: https://issues.apache.org/jira/browse/HDFS-13702 > Project: Hadoop HDFS > Issue Type: Bug > Components: performance >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch > > > I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate > workload even when HTrace is disabled. This is because it stringifies several > integers. We should avoid all allocation and stringification when htrace is > disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled
[
https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525521#comment-16525521
]
Sean Busbey commented on HDFS-13702:
Todd got me to the log
{code}
[INFO] -
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java:[128,30]
error: method newBlockReader in class BlockReaderRemote cannot be applied to
given types;
[INFO] 1 error
[INFO] -
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 26.030 s
[INFO] Finished at: 2018-06-27T00:06:25+00:00
[INFO] Final Memory: 27M/570M
[INFO]
[WARNING] The requested profile "native" could not be activated because it does
not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it
does not exist.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-hdfs: Compilation failure
[ERROR]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java:[128,30]
error: method newBlockReader in class BlockReaderRemote cannot be applied to
given types;
{code}
we've been chatting and it definitely looks like we run against the wrong
artifact. Todd has a plausible theory that maybe a concurrent run of the
snapshot publisher happens to land in the asf snapshot repo between when we did
test-compile and here.
I'm going to rerun in debug.
> HTrace hooks taking 10-15% CPU in DFS client when disabled
> --
>
> Key: HDFS-13702
> URL: https://issues.apache.org/jira/browse/HDFS-13702
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: performance
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Major
> Attachments: hdfs-13702.patch, hdfs-13702.patch, hdfs-13702.patch
>
>
> I am seeing DFSClient.newReaderTraceScope take ~15% CPU in a teravalidate
> workload even when HTrace is disabled. This is because it stringifies several
> integers. We should avoid all allocation and stringification when htrace is
> disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-194) Remove NodePoolManager and node pool handling from SCM
[ https://issues.apache.org/jira/browse/HDDS-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-194: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~elek] for the contribution. I've commit the patch to trunk. > Remove NodePoolManager and node pool handling from SCM > -- > > Key: HDDS-194 > URL: https://issues.apache.org/jira/browse/HDDS-194 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-194.001.patch, HDDS-194.002.patch > > > The current code use NodePoolManager and ContainerSupervisor to group the > nodes to smaller groups (pools) and handle the pull based node reports group > by group. > But this code is not used any more as we switch back to use a push based > model. In the datanode the reports could be handled by the specific report > handlers, and in the scm side the reports will be processed by the > SCMHeartbeatDispatcher which will send the events to the EventQueue. > As of now the NodePool abstraction could be removed from the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525498#comment-16525498 ] Todd Lipcon commented on HDFS-13703: Reattached the same patch to re-run Jenkins > Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit > > > Key: HDFS-13703 > URL: https://issues.apache.org/jira/browse/HDFS-13703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13703.patch, hdfs-13703.patch > > > The DFSClient creates a CorruptedBlocks object, which contains a HashMap, on > every read call. In most cases, a read will not hit any corrupted blocks, and > this hashmap is not used. It seems the JIT isn't smart enough to eliminate > this allocation. We would be better off avoiding it and only allocating in > the rare case when a corrupt block is hit. > Removing this allocation reduced CPU usage of a TeraValidate job by about 10%. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13703: --- Attachment: hdfs-13703.patch > Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit > > > Key: HDFS-13703 > URL: https://issues.apache.org/jira/browse/HDFS-13703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hdfs-13703.patch, hdfs-13703.patch > > > The DFSClient creates a CorruptedBlocks object, which contains a HashMap, on > every read call. In most cases, a read will not hit any corrupted blocks, and > this hashmap is not used. It seems the JIT isn't smart enough to eliminate > this allocation. We would be better off avoiding it and only allocating in > the rare case when a corrupt block is hit. > Removing this allocation reduced CPU usage of a TeraValidate job by about 10%. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-186) Create under replicated queue
[
https://issues.apache.org/jira/browse/HDDS-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525467#comment-16525467
]
genericqa commented on HDDS-186:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDDS-186 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929445/HDDS-186.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9b7e2255c982 4.4.0-121-generic #145-Ubuntu SMP Fri Apr 13
13:47:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fbaff36 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/375/testReport/ |
| Max. process+thread count | 441 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/container-service U: hadoop-hdds/container-service |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/375/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create under replicated queue
> -
>
> Key: HDDS-186
> URL: https://issues.apache.org/jira/browse/HDDS-186
> Project:
[jira] [Commented] (HDDS-194) Remove NodePoolManager and node pool handling from SCM
[ https://issues.apache.org/jira/browse/HDDS-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525466#comment-16525466 ] Xiaoyu Yao commented on HDDS-194: - Thanks [~elek] for working on this. +1 for v2 patch. I will commit it shortly. > Remove NodePoolManager and node pool handling from SCM > -- > > Key: HDDS-194 > URL: https://issues.apache.org/jira/browse/HDDS-194 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-194.001.patch, HDDS-194.002.patch > > > The current code use NodePoolManager and ContainerSupervisor to group the > nodes to smaller groups (pools) and handle the pull based node reports group > by group. > But this code is not used any more as we switch back to use a push based > model. In the datanode the reports could be handled by the specific report > handlers, and in the scm side the reports will be processed by the > SCMHeartbeatDispatcher which will send the events to the EventQueue. > As of now the NodePool abstraction could be removed from the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-200) Create CloseContainerWatcher
Xiaoyu Yao created HDDS-200: --- Summary: Create CloseContainerWatcher Key: HDDS-200 URL: https://issues.apache.org/jira/browse/HDDS-200 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This will be based on HDDS-195. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-182) CleanUp Reimplemented classes
[ https://issues.apache.org/jira/browse/HDDS-182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-182 started by Hanisha Koneru. --- > CleanUp Reimplemented classes > - > > Key: HDDS-182 > URL: https://issues.apache.org/jira/browse/HDDS-182 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > > Cleanup container-service's ozone.container.common package. The following > classes have been refactored and re-implemented. The unused classes/ methods > should be cleaned up. > # org.apache.hadoop.ozone.container.common.impl.Dispatcher > # org.apache.hadoop.ozone.container.common.helpers.ChunkUtils > # org.apache.hadoop.ozone.container.common.helpers.KeyUtils > # org.apache.hadoop.ozone.container.common.impl.ChunkManagerImpl > # org.apache.hadoop.ozone.container.common.impl.KeyManagerImpl -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-182) CleanUp Reimplemented classes
[ https://issues.apache.org/jira/browse/HDDS-182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-182: Description: Cleanup container-service's ozone.container.common package. The following classes have been refactored and re-implemented. The unused classes/ methods should be cleaned up. # org.apache.hadoop.ozone.container.common.impl.Dispatcher # org.apache.hadoop.ozone.container.common.helpers.ChunkUtils # org.apache.hadoop.ozone.container.common.helpers.KeyUtils # org.apache.hadoop.ozone.container.common.impl.ChunkManagerImpl # org.apache.hadoop.ozone.container.common.impl.KeyManagerImpl was: 1. Commands from SCM to Datanode should go through the new HddsDispatcher. 2. Cleanup container-service's ozone.container.common package. > CleanUp Reimplemented classes > - > > Key: HDDS-182 > URL: https://issues.apache.org/jira/browse/HDDS-182 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > > Cleanup container-service's ozone.container.common package. The following > classes have been refactored and re-implemented. The unused classes/ methods > should be cleaned up. > # org.apache.hadoop.ozone.container.common.impl.Dispatcher > # org.apache.hadoop.ozone.container.common.helpers.ChunkUtils > # org.apache.hadoop.ozone.container.common.helpers.KeyUtils > # org.apache.hadoop.ozone.container.common.impl.ChunkManagerImpl > # org.apache.hadoop.ozone.container.common.impl.KeyManagerImpl -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-182) CleanUp Reimplemented classes
[ https://issues.apache.org/jira/browse/HDDS-182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-182: Summary: CleanUp Reimplemented classes (was: Integrate HddsDispatcher) > CleanUp Reimplemented classes > - > > Key: HDDS-182 > URL: https://issues.apache.org/jira/browse/HDDS-182 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > > 1. Commands from SCM to Datanode should go through the new HddsDispatcher. > 2. Cleanup container-service's ozone.container.common package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-183) Integrate Volumeset, ContainerSet and HddsDispatcher
[ https://issues.apache.org/jira/browse/HDDS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-183: Summary: Integrate Volumeset, ContainerSet and HddsDispatcher (was: Integrate Volumeset, ContainerSet.) > Integrate Volumeset, ContainerSet and HddsDispatcher > > > Key: HDDS-183 > URL: https://issues.apache.org/jira/browse/HDDS-183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-183-HDDS-48.00.patch, HDDS-183-HDDS-48.01.patch, > HDDS-183-HDDS-48.02.patch, HDDS-183-HDDS-48.03.patch > > > This Jira adds following: > 1. Use new VolumeSet. > 2. build container map from .container files during startup. > 3. Integrate HddsDispatcher. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13076) [SPS]: Merge work for HDFS-10285
[
https://issues.apache.org/jira/browse/HDFS-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525357#comment-16525357
]
genericqa commented on HDFS-13076:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-13076 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13076 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908186/HDFS-10285-consolidated-merge-patch-01.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24508/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [SPS]: Merge work for HDFS-10285
>
>
> Key: HDFS-13076
> URL: https://issues.apache.org/jira/browse/HDFS-13076
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Priority: Major
> Attachments: HDFS-10285-consolidated-merge-patch-00.patch,
> HDFS-10285-consolidated-merge-patch-01.patch
>
>
> This Jira is to run aggregated HDFS-10285 branch patch against trunk and
> check for any jenkins issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-13485 started by Siyao Meng.
-
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL: https://issues.apache.org/jira/browse/HDFS-13485
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
>Affects Versions: 3.0.0
> Environment: Kerberized. Hadoop 3.0.0, WebHDFS.
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Minor
>
> curl -k -i --negotiate -u : "https://hadoop3-4.example.com:20004/webhdfs/v1;
> DataNode Web UI should do a better error checking/handling.
> {noformat}
> 2018-04-19 10:07:49,338 WARN
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.token.Token.decodeWritable(Token.java:364)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:383)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.ParameterParser.delegationToken(ParameterParser.java:128)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.DataNodeUGIProvider.ugi(DataNodeUGIProvider.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:129)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:51)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:31)
> at
> com.cloudera.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1379)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
>
[jira] [Updated] (HDDS-199) Implement ReplicationManager to replicate ClosedContainers
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-199: -- Summary: Implement ReplicationManager to replicate ClosedContainers (was: Implement ReplicationManager to replicate ClosedContainer) > Implement ReplicationManager to replicate ClosedContainers > -- > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-199) Implement ReplicationManager to replicate ClosedContainer
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-199: - Assignee: Elek, Marton > Implement ReplicationManager to replicate ClosedContainer > - > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-199) Implement ReplicationManager to replicate ClosedContainer
Elek, Marton created HDDS-199: - Summary: Implement ReplicationManager to replicate ClosedContainer Key: HDDS-199 URL: https://issues.apache.org/jira/browse/HDDS-199 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Elek, Marton Fix For: 0.2.1 HDDS/Ozone supports Open and Closed containers. In case of specific conditions (container is full, node is failed) the container will be closed and will be replicated in a different way. The replication of Open containers are handled with Ratis and PipelineManger. The ReplicationManager should handle the replication of the ClosedContainers. The replication information will be sent as an event (UnderReplicated/OverReplicated). The Replication manager will collect all of the events in a priority queue (to replicate first the containers where more replica is missing) calculate the destination datanode (first with a very simple algorithm, later with calculating scatter-width) and send the Copy/Delete container to the datanode (CommandQueue). A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the copy/delete in case of failure. This is an in-memory structure (based on HDDS-195) which can requeue the underreplicated/overreplicated events to the prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-194) Remove NodePoolManager and node pool handling from SCM
[ https://issues.apache.org/jira/browse/HDDS-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525316#comment-16525316 ] Elek, Marton commented on HDDS-194: --- I believe that TestStorageContainerManager.testBlockDeletingThrottling is not related. Here the MiniOzoneCluster can't be started while cluster can be started in all the other tests. TestCloseContainerByPipeline tests the container closing which also seems to be independent. (The test failes where the container state is checked on the datanode. So it's not related to the incoming node reports.) > Remove NodePoolManager and node pool handling from SCM > -- > > Key: HDDS-194 > URL: https://issues.apache.org/jira/browse/HDDS-194 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-194.001.patch, HDDS-194.002.patch > > > The current code use NodePoolManager and ContainerSupervisor to group the > nodes to smaller groups (pools) and handle the pull based node reports group > by group. > But this code is not used any more as we switch back to use a push based > model. In the datanode the reports could be handled by the specific report > handlers, and in the scm side the reports will be processed by the > SCMHeartbeatDispatcher which will send the events to the EventQueue. > As of now the NodePool abstraction could be removed from the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13617) Allow wrapping NN QOP into token in encrypted message
[
https://issues.apache.org/jira/browse/HDFS-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525318#comment-16525318
]
Chen Liang commented on HDFS-13617:
---
Thanks [~xkrogen] for the great comments! I think v002 patch has been rebased.
Any chance you were applying v001 patch? Also, this Jira's patch needs to be
applied on top of HDFS-13566. Dependency link added.
It is great point on including more information into the encrypted message! I
considered client IP address, user name is definitely another good candidate.
Adding more info definitely improves security, but we need to be careful about
what exactly information should be included. As this will depend on whether
this info may change at runtime, whether this info is available at NN rpc
server layer, whether that info is too long, which adds more encryption
overhead etc. I will try to think of all the possibly good candidates and
follow up in next patch. As for now, post v003 patch to address all the other
comments. For {{DFS_QOP_WRAP_HMAC_ALGORITHM_DEFAULT}}, just like you pointed
out, this is hard coded everywhere else, so I simply go with the same way.
> Allow wrapping NN QOP into token in encrypted message
> -
>
> Key: HDFS-13617
> URL: https://issues.apache.org/jira/browse/HDFS-13617
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-13617.001.patch, HDFS-13617.002.patch,
> HDFS-13617.003.patch
>
>
> This Jira allows NN to configurably wrap the QOP it has established with the
> client into the token message sent back to the client. The QOP is sent back
> in encrypted message, using BlockAccessToken encryption key as the key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-193) Make Datanode heartbeat dispatcher in SCM event based
[ https://issues.apache.org/jira/browse/HDDS-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525314#comment-16525314 ] Elek, Marton commented on HDDS-193: --- I believe that TestStorageContainerManager.testBlockDeletingThrottling is not related. Here the MiniOzoneCluster can't be started while cluster can be started in all the other tests. > Make Datanode heartbeat dispatcher in SCM event based > - > > Key: HDDS-193 > URL: https://issues.apache.org/jira/browse/HDDS-193 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-193.001.patch, HDDS-193.002.patch, > HDDS-193.003.patch, HDDS-193.004.patch, HDDS-193.004.patch > > > HDDS-163 introduced a new dispatcher in the SCM side to send the heartbeat > report parts to the appropriate listeners. I propose to make it EventQueue > based to handle/monitor these async calls in the same way as the other events. > Report handlers would subscribe to the specific events to process the > information. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-173) Refactor Dispatcher and implement Handler for new ContainerIO design
[
https://issues.apache.org/jira/browse/HDDS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525312#comment-16525312
]
Hanisha Koneru commented on HDDS-173:
-
Thanks [~xyao].
Fixed the Findbug errors and JUnit test failures except
{{TestStorageContainerManager#testBlockDeletingThrottling}}.
Will fix this and other integration tests together in a separate Jira once all
the related patches are in.
> Refactor Dispatcher and implement Handler for new ContainerIO design
>
>
> Key: HDDS-173
> URL: https://issues.apache.org/jira/browse/HDDS-173
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-173-HDDS-48.001.patch, HDDS-173-HDDS-48.002.patch,
> HDDS-173-HDDS-48.003.patch, HDDS-173-HDDS-48.004.patch
>
>
> Dispatcher will pass the ContainerCommandRequests to the corresponding
> Handler based on the ContainerType. Each ContainerType will have its own
> Handler. The Handler class will process the message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-173) Refactor Dispatcher and implement Handler for new ContainerIO design
[ https://issues.apache.org/jira/browse/HDDS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-173: Attachment: HDDS-173-HDDS-48.004.patch > Refactor Dispatcher and implement Handler for new ContainerIO design > > > Key: HDDS-173 > URL: https://issues.apache.org/jira/browse/HDDS-173 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-173-HDDS-48.001.patch, HDDS-173-HDDS-48.002.patch, > HDDS-173-HDDS-48.003.patch, HDDS-173-HDDS-48.004.patch > > > Dispatcher will pass the ContainerCommandRequests to the corresponding > Handler based on the ContainerType. Each ContainerType will have its own > Handler. The Handler class will process the message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-186) Create under replicated queue
[ https://issues.apache.org/jira/browse/HDDS-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525306#comment-16525306 ] Ajay Kumar commented on HDDS-186: - [~xyao] thanks for review. Addressed them in patch v3. Replaced Long.compare with Integer.compare to avoid casting. > Create under replicated queue > - > > Key: HDDS-186 > URL: https://issues.apache.org/jira/browse/HDDS-186 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-186.00.patch, HDDS-186.01.patch, HDDS-186.02.patch, > HDDS-186.03.patch > > > Create under replicated queue to replicate under replicated containers in > Ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13524) Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
[
https://issues.apache.org/jira/browse/HDFS-13524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDFS-13524:
--
Assignee: Siyao Meng
> Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize
> -
>
> Key: HDFS-13524
> URL: https://issues.apache.org/jira/browse/HDFS-13524
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Major
>
> TestLargeBlock#testLargeBlockSize may fail with error:
> {quote}
> All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:44968,DS-acddd79e-cdf1-4ac5-aac5-e804a2e61600,DISK]]
> are bad. Aborting...
> {quote}
> Tracing back, the error is due to the stress applied to the host sending a
> 2GB block, causing write pipeline ack read timeout:
> {quote}
> 2017-09-10 22:16:07,285 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (DataXceiver.java:writeBlock(742)) - Receiving
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001 src:
> /127.0.0.1:57794 dest: /127.0.0.1:44968
> 2017-09-10 22:16:50,402 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] WARN
> datanode.DataNode (BlockReceiver.java:flushOrSync(434)) - Slow flushOrSync
> took 5383ms (threshold=300ms), isSync:false, flushTotalNanos=5383638982ns,
> volume=file:/tmp/tmp.1oS3ZfDCwq/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/
> 2017-09-10 22:17:54,427 [ResponseProcessor for block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001] WARN
> hdfs.DataStreamer (DataStreamer.java:run(1214)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.net.SocketTimeoutException: 65000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/127.0.0.1:57794 remote=/127.0.0.1:44968]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:434)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
> at
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1104)
> 2017-09-10 22:17:54,432 [DataXceiver for client
> DFSClient_NONMAPREDUCE_998779779_9 at /127.0.0.1:57794 [Receiving block
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001]] INFO
> datanode.DataNode (BlockReceiver.java:receiveBlock(1000)) - Exception for
> BP-682118952-172.26.15.143-1505106964162:blk_1073741825_1001
> java.io.IOException: Connection reset by peer
> {quote}
> Instead of raising read timeout, I suggest increasing cluster size from
> default=1 to 3, so that it has the opportunity to choose a different DN and
> retry.
> Suspect this fails after HDFS-13103, in Hadoop 2.8/3.0.0-alpha1 when we
> introduced client acknowledgement read timeout.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-186) Create under replicated queue
[ https://issues.apache.org/jira/browse/HDDS-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-186: Attachment: HDDS-186.03.patch > Create under replicated queue > - > > Key: HDDS-186 > URL: https://issues.apache.org/jira/browse/HDDS-186 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-186.00.patch, HDDS-186.01.patch, HDDS-186.02.patch, > HDDS-186.03.patch > > > Create under replicated queue to replicate under replicated containers in > Ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDFS-13485:
--
Assignee: Siyao Meng (was: Wei-Chiu Chuang)
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL: https://issues.apache.org/jira/browse/HDFS-13485
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
>Affects Versions: 3.0.0
> Environment: Kerberized. Hadoop 3.0.0, WebHDFS.
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Minor
>
> curl -k -i --negotiate -u : "https://hadoop3-4.example.com:20004/webhdfs/v1;
> DataNode Web UI should do a better error checking/handling.
> {noformat}
> 2018-04-19 10:07:49,338 WARN
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.token.Token.decodeWritable(Token.java:364)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:383)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.ParameterParser.delegationToken(ParameterParser.java:128)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.DataNodeUGIProvider.ugi(DataNodeUGIProvider.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:129)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:51)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:31)
> at
> com.cloudera.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1379)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
>
[jira] [Updated] (HDFS-13617) Allow wrapping NN QOP into token in encrypted message
[ https://issues.apache.org/jira/browse/HDFS-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-13617: -- Attachment: HDFS-13617.003.patch > Allow wrapping NN QOP into token in encrypted message > - > > Key: HDFS-13617 > URL: https://issues.apache.org/jira/browse/HDFS-13617 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13617.001.patch, HDFS-13617.002.patch, > HDFS-13617.003.patch > > > This Jira allows NN to configurably wrap the QOP it has established with the > client into the token message sent back to the client. The QOP is sent back > in encrypted message, using BlockAccessToken encryption key as the key. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDFS-13485:
--
Assignee: Wei-Chiu Chuang
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL: https://issues.apache.org/jira/browse/HDFS-13485
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
>Affects Versions: 3.0.0
> Environment: Kerberized. Hadoop 3.0.0, WebHDFS.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
>
> curl -k -i --negotiate -u : "https://hadoop3-4.example.com:20004/webhdfs/v1;
> DataNode Web UI should do a better error checking/handling.
> {noformat}
> 2018-04-19 10:07:49,338 WARN
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.token.Token.decodeWritable(Token.java:364)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:383)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.ParameterParser.delegationToken(ParameterParser.java:128)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.DataNodeUGIProvider.ugi(DataNodeUGIProvider.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:129)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:51)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:31)
> at
> com.cloudera.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1379)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
>
[jira] [Commented] (HDFS-13076) [SPS]: Merge work for HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525292#comment-16525292 ] Uma Maheswara Rao G commented on HDFS-13076: [~rakeshr] Please use this Jira for merge cleanup task > [SPS]: Merge work for HDFS-10285 > > > Key: HDFS-13076 > URL: https://issues.apache.org/jira/browse/HDFS-13076 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rakesh R >Priority: Major > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch > > > This Jira is to run aggregated HDFS-10285 branch patch against trunk and > check for any jenkins issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13672) clearCorruptLazyPersistFiles could crash NameNode
[
https://issues.apache.org/jira/browse/HDFS-13672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525272#comment-16525272
]
genericqa commented on HDFS-13672:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13672 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929414/HDFS-13672.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b78656d4af6a 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bedc4fe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24507/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24507/testReport/ |
| Max. process+thread count | 3249 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDDS-175) Refactor ContainerInfo to remove Pipeline object from it
[ https://issues.apache.org/jira/browse/HDDS-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525271#comment-16525271 ] Ajay Kumar commented on HDDS-175: - failure in TestContainerStateManager is related to this patch, will fix it with any review comments we have. Other failed test passed locally. > Refactor ContainerInfo to remove Pipeline object from it > - > > Key: HDDS-175 > URL: https://issues.apache.org/jira/browse/HDDS-175 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.2.1 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-175.00.patch, HDDS-175.01.patch, HDDS-175.02.patch, > HDDS-175.03.patch, HDDS-175.04.patch, HDDS-175.05.patch, HDDS-175.06.patch, > HDDS-175.07.patch, HDDS-175.08.patch > > > Refactor ContainerInfo to remove Pipeline object from it. We can add below 4 > fields to ContainerInfo to recreate pipeline if required: > # pipelineId > # replication type > # expected replication count > # DataNode where its replica exist -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
