[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817542#comment-15817542
]
Hadoop QA commented on HDFS-11186:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}144m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.server.namenode.TestListCorruptFileBlocks |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestFileChecksum |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11186 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846741/HDFS-11186-HDFS-10285.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 41b19e50580b 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10285 / 402cf35 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18147/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18147/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[
https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817471#comment-15817471
]
Uma Maheswara Rao G commented on HDFS-11150:
{quote}
I was considering raising a JIRA to fix it because it's part of retry mechanism
and it's not very easy to remove xattar in BlockStorageMovementAttemptedItems
right now.
{quote}
Sure. Agree with you for filing separate JIRA for removing Xattrs. Thanks
> [SPS]: Provide persistence when satisfying storage policy.
> --
>
> Key: HDFS-11150
> URL: https://issues.apache.org/jira/browse/HDFS-11150
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Attachments: HDFS-11150-HDFS-10285.001.patch,
> HDFS-11150-HDFS-10285.002.patch, HDFS-11150-HDFS-10285.003.patch,
> HDFS-11150-HDFS-10285.004.patch, HDFS-11150-HDFS-10285.005.patch,
> HDFS-11150-HDFS-10285.006.patch, editsStored, editsStored.xml
>
>
> Provide persistence for SPS in case that Hadoop cluster crashes by accident.
> Basically we need to change EditLog and FsImage here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817315#comment-15817315
]
Rakesh R commented on HDFS-11186:
-
Thank you [~zhouwei] for the updates. I'm almost OK with the changes. Just few
minor comments, please take care.
# Could you change the word {{inactive}} with proper NN state like,
{code}
throw new ReconfigurationException(property, newVal,
getConf().get(property), new HadoopIllegalArgumentException(
"Activating or deactivating storage policy satisfier service on "
+ state + " NameNode is not allowed"));
{code}
# I just mentioned {{e.printStackTrace();}} as an example in my previous
comment. It would be good to change with proper assertion as shown below:
{code}
} catch (ReconfigurationException e) {
GenericTestUtils.assertExceptionContains("Could not change property "
+ DFSConfigKeys.DFS_STORAGE_POLICY_SATISFIER_ACTIVATE_KEY
+ " from 'true' to 'false'", e);
GenericTestUtils.assertExceptionContains(
"Activating or deactivating storage policy satisfier service on "
+ "standby NameNode is not allowed", e.getCause());
}
{code}
bq. Do you mean that it's better to create another jira to change the timeout
values instead of modifying it in this patch? If so, I'll re-update the patch.
Yeah, since this is not introduced as part of this jira, I'd prefer to raise a
separate task to address the timeout case. Please remove those changes from
this patch.
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch,
> HDFS-11186-HDFS-10285.03.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10733) NameNode terminated after full GC thinking QJM is unresponsive.
[
https://issues.apache.org/jira/browse/HDFS-10733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817204#comment-15817204
]
Hadoop QA commented on HDFS-10733:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846719/HDFS-10733.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9e39f19c6ee1 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4db119b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18142/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18142/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18142/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> NameNode terminated after full GC thinking QJM is unresponsive.
> ---
>
> Key: HDFS-10733
> URL: https://issues.apache.org/jira/browse/HDFS-10733
> Project: Hadoop HDFS
> Issue Type: Bug
>
[jira] [Commented] (HDFS-9391) Update webUI/JMX to display maintenance state info
[
https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817184#comment-15817184
]
Manoj Govindassamy commented on HDFS-9391:
--
* A clean build with the latest pull for both trunk and branch-2 passes through
for me on my local system.
* Even in this Jenkins Failure log, Build succeeded, and later the deploy part
failed due to a tool missing.
{noformat}
4591 [INFO] Apache Hadoop Client Modules ... SUCCESS [
0.335 s]
4592 [INFO]
4593 [INFO] BUILD SUCCESS
4594 [INFO]
4595 [INFO] Total time: 07:26 min (Wall Clock)
4596 [INFO] Finished at: 2017-01-11T04:38:09+00:00
4597 [INFO] Final Memory: 315M/4892M
4598 [INFO]
4599 + /home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn deploy
-DdeployAtEnd=true -DretryFailedDeploymentCount=10 -DskipTests
-Dmaven.repo.local=/home/jenkins/jenkins-slave
/workspace/Hadoop-trunk-Commit/maven-repo
4600 [INFO] Scanning for projects...
..
..
5217 Build step 'Execute shell' marked build as failure
5218 Updating HDFS-9391
5219 ERROR: No tool found matching LATEST1_8_HOME
5220 Setting MAVEN_3_3_3_HOME=/home/jenkins/tools/maven/apache-maven-3.3.3
5221 Finished: FAILURE
{noformat}
> Update webUI/JMX to display maintenance state info
> --
>
> Key: HDFS-9391
> URL: https://issues.apache.org/jira/browse/HDFS-9391
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Ming Ma
>Assignee: Manoj Govindassamy
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-9391-MaintenanceMode-WebUI.pdf,
> HDFS-9391-branch-2-MaintenanceMode-WebUI.pdf, HDFS-9391-branch-2.01.patch,
> HDFS-9391-branch-2.02.patch, HDFS-9391.01.patch, HDFS-9391.02.patch,
> HDFS-9391.03.patch, HDFS-9391.04.patch, Maintenance webUI.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9391) Update webUI/JMX to display maintenance state info
[ https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817174#comment-15817174 ] Manoj Govindassamy commented on HDFS-9391: -- Thanks [~mingma] for the review and commit. Much appreciated. > Update webUI/JMX to display maintenance state info > -- > > Key: HDFS-9391 > URL: https://issues.apache.org/jira/browse/HDFS-9391 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Ming Ma >Assignee: Manoj Govindassamy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HDFS-9391-MaintenanceMode-WebUI.pdf, > HDFS-9391-branch-2-MaintenanceMode-WebUI.pdf, HDFS-9391-branch-2.01.patch, > HDFS-9391-branch-2.02.patch, HDFS-9391.01.patch, HDFS-9391.02.patch, > HDFS-9391.03.patch, HDFS-9391.04.patch, Maintenance webUI.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11186) [SPS]: Daemon thread of SPS should start only in Active NN
[
https://issues.apache.org/jira/browse/HDFS-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhou updated HDFS-11186:
Attachment: HDFS-11186-HDFS-10285.03.patch
Thanks [~rakeshr] for reviewing the patch!
{quote}
Probably, we could observe the impact of this additional thread.join(3000)
timeout and raise a separate improvement task(if any changes needed).
{quote}
Do you mean that it's better to create another jira to change the timeout
values instead of modifying it in this patch? If so, I'll re-update the patch.
Thanks!
> [SPS]: Daemon thread of SPS should start only in Active NN
> --
>
> Key: HDFS-11186
> URL: https://issues.apache.org/jira/browse/HDFS-11186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: HDFS-11186-HDFS-10285.00.patch,
> HDFS-11186-HDFS-10285.01.patch, HDFS-11186-HDFS-10285.02.patch,
> HDFS-11186-HDFS-10285.03.patch
>
>
> As discussed in [HDFS-10885
> |https://issues.apache.org/jira/browse/HDFS-10885], we need to ensure that
> SPS is started only in Active NN. This JIRA is opened for discussion and
> tracking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9391) Update webUI/JMX to display maintenance state info
[ https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817144#comment-15817144 ] Hudson commented on HDFS-9391: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11104 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11104/]) HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj (mingma: rev 467f5f1735494c5ef74e6591069884d3771c17e4) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java > Update webUI/JMX to display maintenance state info > -- > > Key: HDFS-9391 > URL: https://issues.apache.org/jira/browse/HDFS-9391 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Ming Ma >Assignee: Manoj Govindassamy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HDFS-9391-MaintenanceMode-WebUI.pdf, > HDFS-9391-branch-2-MaintenanceMode-WebUI.pdf, HDFS-9391-branch-2.01.patch, > HDFS-9391-branch-2.02.patch, HDFS-9391.01.patch, HDFS-9391.02.patch, > HDFS-9391.03.patch, HDFS-9391.04.patch, Maintenance webUI.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11121) Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping
[
https://issues.apache.org/jira/browse/HDFS-11121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817127#comment-15817127
]
Wei-Chiu Chuang commented on HDFS-11121:
Sorry I missed your update. Will follow up soon.
> Add assertions to BlockInfo#addStorage to protect from breaking
> reportedBlock-blockGroup mapping
>
>
> Key: HDFS-11121
> URL: https://issues.apache.org/jira/browse/HDFS-11121
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11121.1.patch, HDFS-11121.2.patch
>
>
> There are not any assertions in {{BlockInfo.addStorage}}. This may cause that
> {{BlockInfo}} instances accept strange block reports and result in serious
> bugs, like HDFS-10858.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-9391) Update webUI/JMX to display maintenance state info
[ https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817119#comment-15817119 ] Ming Ma edited comment on HDFS-9391 at 1/11/17 4:23 AM: Thanks [~manojg] for the contribution. Thanks [~dilaver] and [~eddyxu] for the review. I have committed the patch to trunk and branch-2. was (Author: mingma): Thanks [~manojg] for the contribution. Thanks [~eddyxu] for the review. I have committed the patch to trunk and branch-2. > Update webUI/JMX to display maintenance state info > -- > > Key: HDFS-9391 > URL: https://issues.apache.org/jira/browse/HDFS-9391 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Ming Ma >Assignee: Manoj Govindassamy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HDFS-9391-MaintenanceMode-WebUI.pdf, > HDFS-9391-branch-2-MaintenanceMode-WebUI.pdf, HDFS-9391-branch-2.01.patch, > HDFS-9391-branch-2.02.patch, HDFS-9391.01.patch, HDFS-9391.02.patch, > HDFS-9391.03.patch, HDFS-9391.04.patch, Maintenance webUI.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9391) Update webUI/JMX to display maintenance state info
[ https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9391: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Thanks [~manojg] for the contribution. Thanks [~eddyxu] for the review. I have committed the patch to trunk and branch-2. > Update webUI/JMX to display maintenance state info > -- > > Key: HDFS-9391 > URL: https://issues.apache.org/jira/browse/HDFS-9391 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Ming Ma >Assignee: Manoj Govindassamy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HDFS-9391-MaintenanceMode-WebUI.pdf, > HDFS-9391-branch-2-MaintenanceMode-WebUI.pdf, HDFS-9391-branch-2.01.patch, > HDFS-9391-branch-2.02.patch, HDFS-9391.01.patch, HDFS-9391.02.patch, > HDFS-9391.03.patch, HDFS-9391.04.patch, Maintenance webUI.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817108#comment-15817108
]
Hadoop QA commented on HDFS-11296:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 34s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_111 Failed junit tests |
hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker |
| JDK v1.7.0_121 Failed junit tests | hadoop.hdfs.TestAclsEndToEnd |
| | hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
|
[jira] [Commented] (HDFS-11191) Datanode Capacity is misleading if the dfs.datanode.data.dir is configured with two directories from the same file system.
[
https://issues.apache.org/jira/browse/HDFS-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817066#comment-15817066
]
Weiwei Yang commented on HDFS-11191:
Can someone help to review the v7 patch?
The idea is: Added a string field {{fileSystem}} in {{DatanodeStorage}}, when
datanode detects volume changes, it gets the file system where the volume
mounted to, then keep that in {{DatanodeStorage}} and send that to namenode in
heartbeat. Note get file system call in {{DU}} is a heavy linux call that won't
be called in every heartbeat. When namenode knows which file system each
datanode storage belongs to, it can avoid counting same file system capacity
twice when calculating datanode capacity. The default value of storage
{{fileSystem}} is an EMPTY string, I added an internal configuration property
to disable this for testing (as mini cluster is supposed to always run with
single disk).
> Datanode Capacity is misleading if the dfs.datanode.data.dir is configured
> with two directories from the same file system.
> --
>
> Key: HDFS-11191
> URL: https://issues.apache.org/jira/browse/HDFS-11191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.5.0
> Environment: SLES 11SP3
> HDP 2.5.0
>Reporter: Deepak Chander
>Assignee: Weiwei Yang
> Labels: capacity, datanode, storage, user-experience
> Attachments: HDFS-11191.01.patch, HDFS-11191.02.patch,
> HDFS-11191.03.patch, HDFS-11191.04.patch, HDFS-11191.05.patch,
> HDFS-11191.06.patch, HDFS-11191.07.patch
>
>
> In the command “hdfs dfsadmin -report” The Configured Capacity is misleading
> if the dfs.datanode.data.dir is configured with two directories from the same
> file system.
> hdfs@kimtest1:~> hdfs dfsadmin -report
> Configured Capacity: 239942369274 (223.46 GB)
> Present Capacity: 207894724602 (193.62 GB)
> DFS Remaining: 207894552570 (193.62 GB)
> DFS Used: 172032 (168 KB)
> DFS Used%: 0.00%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> -
> Live datanodes (3):
> Name: 172.26.79.87:50010 (kimtest3)
> Hostname: kimtest3
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 9528000512 (8.87 GB)
> DFS Remaining: 70452731902 (65.61 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.09%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> Name: 172.26.80.38:50010 (kimtest4)
> Hostname: kimtest4
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 13010952192 (12.12 GB)
> DFS Remaining: 66969780222 (62.37 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 83.73%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> Name: 172.26.79.86:50010 (kimtest2)
> Hostname: kimtest2
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 9508691968 (8.86 GB)
> DFS Remaining: 70472040446 (65.63 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.11%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> If you see my datanode root file system size its only 38GB
> kimtest3:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 2.6G 33G 8% /
> kimtest4:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 4.2G 32G 12% /
> kimtest2:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 2.6G 33G 8% /
> The below is from hdfs-site.xml file
>
> dfs.datanode.data.dir
> file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn
>
> I have removed the other directory grid1 and restarted datanode process.
>
> dfs.datanode.data.dir
> file:///grid/hadoop/hdfs/dn
>
> Now the size is reflecting correctly
> hdfs@kimtest1:/grid> hdfs dfsadmin -report
> Configured Capacity: 119971184637 (111.73 GB)
> Present Capacity: 103947243517 (96.81 GB)
> DFS Remaining: 103947157501 (96.81 GB)
> DFS Used: 86016 (84 KB)
> DFS Used%: 0.00%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with
[jira] [Commented] (HDFS-10530) BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy
[
https://issues.apache.org/jira/browse/HDFS-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817065#comment-15817065
]
Hadoop QA commented on HDFS-10530:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
43s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 372 unchanged - 0 fixed = 374 total (was 372) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
43s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10530 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12815405/HDFS-10530.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 991eb2ce5c51 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4db119b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/18144/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18144/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/18144/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18144/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/18144/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/18144/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size
[
https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817063#comment-15817063
]
Hadoop QA commented on HDFS-9342:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-9342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12779616/HDFS-9342.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5a69cc5965b5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4db119b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18140/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18140/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure coding: client should update and commit block based on acknowledged
> size
>
>
> Key: HDFS-9342
> URL: https://issues.apache.org/jira/browse/HDFS-9342
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Zhe Zhang
>Assignee: Walter Su
> Labels:
[jira] [Commented] (HDFS-11312) Discrepancy in nonDfsUsed index in protobuf
[
https://issues.apache.org/jira/browse/HDFS-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817054#comment-15817054
]
Hadoop QA commented on HDFS-11312:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11312 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846701/HDFS-11312.001.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 15272b345031 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4db119b |
| Default Java | 1.8.0_111 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18143/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18143/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Discrepancy in nonDfsUsed index in protobuf
> ---
>
> Key: HDFS-11312
> URL: https://issues.apache.org/jira/browse/HDFS-11312
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Blocker
> Attachments: HDFS-11312.001.patch
>
>
> The patches for HDFS-9038 had a discrepancy between trunk and branch-2.7: in
> one message type, nonDfsUsed is given 2 different indices. This is a minor
> wire incompatibility that is easy to fix...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9822) Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped block at the same time
[
https://issues.apache.org/jira/browse/HDFS-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817023#comment-15817023
]
Hadoop QA commented on HDFS-9822:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HDFS-9822 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-9822 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791223/HDFS-9822-002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18146/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped
> block at the same time
>
>
> Key: HDFS-9822
> URL: https://issues.apache.org/jira/browse/HDFS-9822
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Rakesh R
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9822-001.patch, HDFS-9822-002.patch
>
>
> Found the following AssertionError in
> https://builds.apache.org/job/PreCommit-HDFS-Build/14501/testReport/org.apache.hadoop.hdfs.server.namenode/TestReconstructStripedBlocks/testMissingStripedBlockWithBusyNode2/
> {code}
> AssertionError: Should wait the previous reconstruction to finish
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.validateReconstructionWork(BlockManager.java:1680)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReconstructionWorkForBlocks(BlockManager.java:1536)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockReconstructionWork(BlockManager.java:1472)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:4229)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:4100)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:4119)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9391) Update webUI/JMX to display maintenance state info
[
https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817020#comment-15817020
]
Hadoop QA commented on HDFS-9391:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 294 unchanged - 1 fixed = 294 total (was 295) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 51m
12s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_111 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-9391 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846716/HDFS-9391-branch-2.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 56a007f022c0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / ce5ad0e |
|
[jira] [Commented] (HDFS-9657) Schedule EC tasks at proper time to reduce the impact of recovery traffic
[
https://issues.apache.org/jira/browse/HDFS-9657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817022#comment-15817022
]
Hadoop QA commented on HDFS-9657:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-9657 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-9657 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785958/HDFS-9657-002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18145/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Schedule EC tasks at proper time to reduce the impact of recovery traffic
> -
>
> Key: HDFS-9657
> URL: https://issues.apache.org/jira/browse/HDFS-9657
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Li Bo
>Assignee: Li Bo
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9657-001.patch, HDFS-9657-002.patch
>
>
> The EC recover tasks consume a lot of network bandwidth and disk I/O.
> Recovering a corrupt block requires transferring 6 blocks , hence creating a
> 6X overhead in network bandwidth and disk I/O. When a datanode fails , the
> recovery of the whole blocks on this datanode may use up the network
> bandwith. We need to start a recovery task at a proper time in order to give
> less impact to the system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11306) Print remaining edit logs from buffer if edit log can't be rolled.
[
https://issues.apache.org/jira/browse/HDFS-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816995#comment-15816995
]
Hadoop QA commented on HDFS-11306:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11306 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846724/HDFS-11306.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 69289742c7af 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e692316 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18139/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18139/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18139/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18139/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Print remaining edit logs from buffer if edit log can't be rolled.
> --
>
>
[jira] [Updated] (HDFS-11312) Discrepancy in nonDfsUsed index in protobuf
[ https://issues.apache.org/jira/browse/HDFS-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11312: Status: Patch Available (was: Open) [~mackrorysd] thanks for reporting this and providing the patch..Patch LGTM..Pending for jenkins and [~arpitagarwal] have a look on this.. > Discrepancy in nonDfsUsed index in protobuf > --- > > Key: HDFS-11312 > URL: https://issues.apache.org/jira/browse/HDFS-11312 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha1, 2.8.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Blocker > Attachments: HDFS-11312.001.patch > > > The patches for HDFS-9038 had a discrepancy between trunk and branch-2.7: in > one message type, nonDfsUsed is given 2 different indices. This is a minor > wire incompatibility that is easy to fix... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11313) Segmented Block Reports
[ https://issues.apache.org/jira/browse/HDFS-11313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816933#comment-15816933 ] Vinitha Reddy Gankidi commented on HDFS-11313: -- [~shv] This idea seems promising. I would like to work on it. I wanted to note that HDFS-7923 is related in the sense that the blocks reports by the DN are sent only when the NN gives the signal. Even with this patch, the issue of processing large DN reports under a global namespace lock still remains. > Segmented Block Reports > --- > > Key: HDFS-11313 > URL: https://issues.apache.org/jira/browse/HDFS-11313 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.2 >Reporter: Konstantin Shvachko > > Block reports from a single DataNode can be currently split into multiple > RPCs each reporting a single DataNode storage (disk). The reports are still > large since disks are getting bigger. Splitting blockReport RPCs into > multiple smaller calls would improve NameNode performance and overall HDFS > stability. > This was discussed in multiple jiras. Here the approach is to let NameNode > divide blockID space into segments and then ask DataNodes to report replicas > in a particular range of IDs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11170) Add create API in filesystem public class to support assign parameter through builder
[
https://issues.apache.org/jira/browse/HDFS-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhou updated HDFS-11170:
Attachment: HDFS-11170-01.patch
Hi [~andrew.wang], thanks for reviewing the patch and the great suggestions!
The patch is updated with some issues:
{code}
a factory method in FileSystem to get the FS-specific builder?
{code}
I added a function {{newCreateBuilder}} in {{FileSystem}} to generate specific
file system create builder, I'm not sure it proper to do it in this way or not.
{code}
add a new private DFS#create that takes checksumOpt and flags along with
favoredNodes is set.
{code}
Do you mean all the {{create}} functions in {{DistributedFileSystem}} goes down
to the same private DFS#create? If so, I found there are some difference in
implementation between {{create(..., final InetSocketAddress[] favoredNodes)}}
and {{create(..., final ChecksumOpt checksumOpt)}}.
Thanks again!
> Add create API in filesystem public class to support assign parameter through
> builder
> -
>
> Key: HDFS-11170
> URL: https://issues.apache.org/jira/browse/HDFS-11170
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: SammiChen
>Assignee: Wei Zhou
> Attachments: HDFS-11170-00.patch, HDFS-11170-01.patch
>
>
> FileSystem class supports multiple create functions to help user create file.
> Some create functions has many parameters, it's hard for user to exactly
> remember these parameters and their orders. This task is to add builder
> based create functions to help user more easily create file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HDFS-8095.
---
Resolution: Not A Problem
Resolving per above comments, since we think that this can be handled by a
combination of HDFS-7859 and HDFS-11314. Thanks [~drankye] for the discussion!
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816879#comment-15816879
]
Andrew Wang commented on HDFS-8095:
---
Cool, sounds good :) I filed HDFS-11314 to track this validation, I guess we
can resolve this JIRA then as dupe/not a problem.
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11314) Validate client-provided EC schema on the NameNode
Andrew Wang created HDFS-11314: -- Summary: Validate client-provided EC schema on the NameNode Key: HDFS-11314 URL: https://issues.apache.org/jira/browse/HDFS-11314 Project: Hadoop HDFS Issue Type: Sub-task Components: erasure-coding Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Filing based on discussion in HDFS-8095. A user might specify a policy that is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster only has 10 nodes. The NN should only allow the client to choose from a pre-approved list determined by the cluster administrator. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8031) Follow-on work for erasure coding phase I (striping layout)
[ https://issues.apache.org/jira/browse/HDFS-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816870#comment-15816870 ] Andrew Wang commented on HDFS-8031: --- Hi folks, I've gone through the 78 open subtasks and added the "hdfs-ec-3.0-nice-to-have" and "hdfs-ec-3.0-must-do" labels as I thought appropriate. I think there's also room for splitting out additional umbrellas, since the number of subtasks on this one is getting unmanageable. There seem to be a few themes: * conversion between replication and EC * performance * coder implementation, configuration, and usage * additional testing There are also additional misc new features and bugs. I propose we move out the longer-term features that aren't immediately being worked on to separate JIRAs (e.g. the truncate/append work, LRC and pluggable block placement). I think we should also move out bugs, and add at least the "nice-to-have" if not "must-do" label for tracking purposes. Overall the goal here is to get the # of subtasks down to a manageable number, so we can actually resolve this umbrella in time for Hadoop 3 GA. If this sounds good, I'll go ahead and reshuffle the JIRAs later this week. Thanks! > Follow-on work for erasure coding phase I (striping layout) > --- > > Key: HDFS-8031 > URL: https://issues.apache.org/jira/browse/HDFS-8031 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFSErasureCodingSystemTestReport-20151106.pdf, > HDFSErasureCodingSystemTestReport-20151106.v2.pdf > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10530) BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy
[ https://issues.apache.org/jira/browse/HDFS-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10530: --- Labels: hdfs-ec-3.0-nice-to-have (was: ) > BlockManager reconstruction work scheduling should correctly adhere to EC > block placement policy > > > Key: HDFS-10530 > URL: https://issues.apache.org/jira/browse/HDFS-10530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Rui Gao >Assignee: Rui Gao > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-10530.1.patch, HDFS-10530.2.patch, > HDFS-10530.3.patch > > > This issue was found by [~tfukudom]. > Under RS-DEFAULT-6-3-64k EC policy, > 1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each) > of the cluster. > 2. Reconstruction work would be scheduled if the 6th rack is added. > 3. While adding the 7th rack or more racks will not trigger reconstruction > work. > Based on default EC block placement policy defined in > “BlockPlacementPolicyRackFaultTolerant.java”, EC file should be able to be > scheduled to distribute to 9 racks if possible. > In *BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock)* , > *numReplicas* of striped blocks might should be *getRealTotalBlockNum()*, > instead of *getRealDataBlockNum()*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11124) Report blockIds of internal blocks for EC files in Fsck
[
https://issues.apache.org/jira/browse/HDFS-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11124:
---
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Report blockIds of internal blocks for EC files in Fsck
> ---
>
> Key: HDFS-11124
> URL: https://issues.apache.org/jira/browse/HDFS-11124
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11124.1.patch
>
>
> At moment, when we do fsck for an EC file which has corrupt blocks and
> missing blocks, the result of fsck is like this:
> {quote}
> /data/striped 393216 bytes, erasure-coded: policy=RS-DEFAULT-6-3-64k, 1
> block(s):
> /data/striped: CORRUPT blockpool BP-1204772930-172.16.165.209-1478761131832
> block blk_-9223372036854775792
> CORRUPT 1 blocks of total size 393216 B
> 0. BP-1204772930-172.16.165.209-1478761131832:blk_-9223372036854775792_1001
> len=393216 Live_repl=4
> [DatanodeInfoWithStorage[127.0.0.1:61617,DS-bcfebe1f-ff54-4d57-9258-ff5bdfde01b5,DISK](CORRUPT),
>
> DatanodeInfoWithStorage[127.0.0.1:61601,DS-9abf64d0-bb6b-434c-8c5e-de8e3b278f91,DISK](CORRUPT),
>
> DatanodeInfoWithStorage[127.0.0.1:61596,DS-62698e61-c13f-44f2-9da5-614945960221,DISK](CORRUPT),
>
> DatanodeInfoWithStorage[127.0.0.1:61605,DS-bbce6708-16fe-44ca-9f1c-506cf00f7e0d,DISK](LIVE),
>
> DatanodeInfoWithStorage[127.0.0.1:61592,DS-9cdd4afd-2dc8-40da-8805-09712e2afcc4,DISK](LIVE),
>
> DatanodeInfoWithStorage[127.0.0.1:61621,DS-f2a72d28-c880-4ffe-a70f-0f403e374504,DISK](LIVE),
>
> DatanodeInfoWithStorage[127.0.0.1:61629,DS-fa6ac558-2c38-41fe-9ef8-222b3f6b2b3c,DISK](LIVE)]
> {quote}
> It would be useful for admins if it reports the blockIds of the internal
> blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11121) Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping
[
https://issues.apache.org/jira/browse/HDFS-11121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11121:
---
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Add assertions to BlockInfo#addStorage to protect from breaking
> reportedBlock-blockGroup mapping
>
>
> Key: HDFS-11121
> URL: https://issues.apache.org/jira/browse/HDFS-11121
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11121.1.patch, HDFS-11121.2.patch
>
>
> There are not any assertions in {{BlockInfo.addStorage}}. This may cause that
> {{BlockInfo}} instances accept strange block reports and result in serious
> bugs, like HDFS-10858.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11072) Add ability to unset and change directory EC policy
[ https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816856#comment-15816856 ] SammiChen commented on HDFS-11072: -- Thanks Andrew, for all the reviewing efforts and commit! > Add ability to unset and change directory EC policy > --- > > Key: HDFS-11072 > URL: https://issues.apache.org/jira/browse/HDFS-11072 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-11072-v1.patch, HDFS-11072-v2.patch, > HDFS-11072-v3.patch, HDFS-11072-v4.patch, HDFS-11072-v5.patch, > HDFS-11072-v6.patch, HDFS-11072-v7.patch > > > Since the directory-level EC policy simply applies to files at create time, > it makes sense to make it more similar to storage policies and allow changing > and unsetting the policy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11066) Improve test coverage for Java coder/ISA-L native coder
[ https://issues.apache.org/jira/browse/HDFS-11066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11066: --- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Improve test coverage for Java coder/ISA-L native coder > --- > > Key: HDFS-11066 > URL: https://issues.apache.org/jira/browse/HDFS-11066 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang > Labels: hdfs-ec-3.0-nice-to-have > > HDFS-10935 found a bug that was only exposed without using native ISA-L > library. We should improve test coverage for both Java/native codec, and even > for mixed scenario (e.g. some nodes use Java codec while the other use native > codec) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10531) Add EC policy and storage policy related usage summarization function to dfs du command
[
https://issues.apache.org/jira/browse/HDFS-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-10531:
---
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Add EC policy and storage policy related usage summarization function to dfs
> du command
> ---
>
> Key: HDFS-10531
> URL: https://issues.apache.org/jira/browse/HDFS-10531
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Rui Gao
>Assignee: Wei-Chiu Chuang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-10531.001.patch
>
>
> Currently du command output:
> {code}
> [ ~]$ hdfs dfs -du -h /home/rgao/
> 0 /home/rgao/.Trash
> 0 /home/rgao/.staging
> 100 M /home/rgao/ds
> 250 M /home/rgao/ds-2
> 200 M /home/rgao/noECBackup-ds
> 500 M /home/rgao/noECBackup-ds-2
> {code}
> For hdfs users and administrators, EC policy and storage policy related usage
> summarization would be very helpful when managing storages of cluster. The
> imitate output of du could be like the following.
> {code}
> [ ~]$ hdfs dfs -du -h -t( total, parameter to be added) /home/rgao
>
> 0 /home/rgao/.Trash
> 0 /home/rgao/.staging
> [Archive] [EC:RS-DEFAULT-6-3-64k] 100 M /home/rgao/ds
> [DISK] [EC:RS-DEFAULT-6-3-64k] 250 M /home/rgao/ds-2
> [DISK] [Replica] 200 M /home/rgao/noECBackup-ds
> [DISK] [Replica] 500 M /home/rgao/noECBackup-ds-2
>
> Total:
>
> [Archive][EC:RS-DEFAULT-6-3-64k] 100 M
> [Archive][Replica]0 M
> [DISK] [EC:RS-DEFAULT-6-3-64k] 250 M
> [DISK] [Replica] 700 M
>
> [Archive][ALL] 100M
> [DISK][ALL] 950M
> [ALL] [EC:RS-DEFAULT-6-3-64k]350M
> [ALL] [Replica] 700M
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10983: --- Labels: hdfs-ec-3.0-nice-to-have (was: ) > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. > Not just webhdfs, but delimiter output does not print if a file is stripped > or not either. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11209) SNN can't checkpoint when rolling upgrade is not finalized
[
https://issues.apache.org/jira/browse/HDFS-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11209:
--
Description:
Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432
brings this back.
With HDFS-8432, the primary NN will not update the VERSION file to the new
version after running with "rollingUpgrade" option until upgrade is finalized.
This is to support more downgrade use cases.
However, the checkpoint on the SNN is incorrectly updating the VERSION file
when the rollingUpgrade is not finalized yet on the primary NN. As a result,
the SNN checkpoint successfully but fail to push it to the primary NN because
its version is higher than the primary NN as shown below.
{code}
2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode
(SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException:
Image uploading failed, status: 403, url:
http://NN:50070/imagetransfer?txid=345404754=IMAGE...,
message: This namenode has storage info -60:221856466:1444080250181:clusterX
but the secondary expected -63:221856466:1444080250181:clusterX
{code}
was:
Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432
brings this back.
With HDFS-8432, the primary NN will not update the VERSION file to the new
version after running with "rollingUpgrade" option until upgrade is finalized.
This is to support more downgrade use cases.
However, the checkpoint on the SNN is incorrectly updating the VERSION file
when the rollingUpgrade is not finalized yet. As a result, the SNN checkpoint
successfully but fail to push it to the primary NN because its version is
higher than the primary NN as shown below.
{code}
2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode
(SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException:
Image uploading failed, status: 403, url:
http://NN:50070/imagetransfer?txid=345404754=IMAGE...,
message: This namenode has storage info -60:221856466:1444080250181:clusterX
but the secondary expected -63:221856466:1444080250181:clusterX
{code}
> SNN can't checkpoint when rolling upgrade is not finalized
> --
>
> Key: HDFS-11209
> URL: https://issues.apache.org/jira/browse/HDFS-11209
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Critical
> Attachments: HDFS-11209.00.patch, HDFS-11209.01.patch,
> HDFS-11209.02.patch, HDFS-11209.03.patch
>
>
> Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432
> brings this back.
> With HDFS-8432, the primary NN will not update the VERSION file to the new
> version after running with "rollingUpgrade" option until upgrade is
> finalized. This is to support more downgrade use cases.
> However, the checkpoint on the SNN is incorrectly updating the VERSION file
> when the rollingUpgrade is not finalized yet on the primary NN. As a result,
> the SNN checkpoint successfully but fail to push it to the primary NN because
> its version is higher than the primary NN as shown below.
> {code}
> 2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode
> (SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException:
> Image uploading failed, status: 403, url:
> http://NN:50070/imagetransfer?txid=345404754=IMAGE...,
> message: This namenode has storage info -60:221856466:1444080250181:clusterX
> but the secondary expected -63:221856466:1444080250181:clusterX
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9962) Erasure Coding: need a way to test multiple EC policies
[ https://issues.apache.org/jira/browse/HDFS-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816851#comment-15816851 ] Andrew Wang commented on HDFS-9962: --- It looks like we still lack complete test coverage, I see that RS (10,4), RS (6,3), and XOR 2,1 are covered by subclasses of TestDFSStripedInputStream but we're still missing RS (3,2). Not sure how comprehensive other tests are either. > Erasure Coding: need a way to test multiple EC policies > --- > > Key: HDFS-9962 > URL: https://issues.apache.org/jira/browse/HDFS-9962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Labels: hdfs-ec-3.0-nice-to-have > > Now that we support multiple EC policies, we need a way test it to catch > potential issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816849#comment-15816849
]
Kai Zheng commented on HDFS-8095:
-
bq. When you say data corruption, do you mean data loss if the cluster has
insufficient # datanodes or racks?
What I meant is, changing of the system default policy may impact big. But if
we can ensure the policy ID is always associated with the file and persisted in
meta, it should be ok even after the system default policy has changed.
Your understanding reminded me that the user specified policy may be not
validated for the use, as you said, there may be no enough datanodes or racks.
So I thought it's a good idea we should add some necessary check in NN side
when user specifying a policy for a file/folder, or whenever a policy is newly
referenced by client requests. We need a separate issue to handle this guess?
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9962) Erasure Coding: need a way to test multiple EC policies
[ https://issues.apache.org/jira/browse/HDFS-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-9962: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Erasure Coding: need a way to test multiple EC policies > --- > > Key: HDFS-9962 > URL: https://issues.apache.org/jira/browse/HDFS-9962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Labels: hdfs-ec-3.0-nice-to-have > > Now that we support multiple EC policies, we need a way test it to catch > potential issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9822) Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped block at the same time
[
https://issues.apache.org/jira/browse/HDFS-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816843#comment-15816843
]
Andrew Wang commented on HDFS-9822:
---
This one looks like a good bug, adding to "nice-to-have" and it might get
upgrade to "must-do". [~rakeshr] do you still have plans to work on it?
> Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped
> block at the same time
>
>
> Key: HDFS-9822
> URL: https://issues.apache.org/jira/browse/HDFS-9822
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Rakesh R
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9822-001.patch, HDFS-9822-002.patch
>
>
> Found the following AssertionError in
> https://builds.apache.org/job/PreCommit-HDFS-Build/14501/testReport/org.apache.hadoop.hdfs.server.namenode/TestReconstructStripedBlocks/testMissingStripedBlockWithBusyNode2/
> {code}
> AssertionError: Should wait the previous reconstruction to finish
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.validateReconstructionWork(BlockManager.java:1680)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReconstructionWorkForBlocks(BlockManager.java:1536)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockReconstructionWork(BlockManager.java:1472)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:4229)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:4100)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:4119)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9822) Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped block at the same time
[
https://issues.apache.org/jira/browse/HDFS-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9822:
--
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped
> block at the same time
>
>
> Key: HDFS-9822
> URL: https://issues.apache.org/jira/browse/HDFS-9822
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Rakesh R
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9822-001.patch, HDFS-9822-002.patch
>
>
> Found the following AssertionError in
> https://builds.apache.org/job/PreCommit-HDFS-Build/14501/testReport/org.apache.hadoop.hdfs.server.namenode/TestReconstructStripedBlocks/testMissingStripedBlockWithBusyNode2/
> {code}
> AssertionError: Should wait the previous reconstruction to finish
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.validateReconstructionWork(BlockManager.java:1680)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReconstructionWorkForBlocks(BlockManager.java:1536)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockReconstructionWork(BlockManager.java:1472)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:4229)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:4100)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:4119)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11313) Segmented Block Reports
[ https://issues.apache.org/jira/browse/HDFS-11313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816841#comment-15816841 ] Konstantin Shvachko commented on HDFS-11313: Full block report (FBR) processing on the NameNode does four things: # Update replica information reported by the DataNode for known blocks # Add new replicas reported by the DataNode # Instruct the DataNode to delete replicas, which belong to non-existing blocks # Remove replicas, which NameNode assumed to be present on the DataNode, but which did not appear in the report The main problem with current FBRs is that they are processed under global namesystem lock, and since the reports are big, other operation cannot proceed until the lock is released. On large clusters the current trend is to decrease FBR frequency, sending FBRs once in 6, 10, or even 12 hours. It would be beneficial to split FBRs into smaller even though more frequent RPC calls. If a DataNode were to split its FBR into multiple RPCs arbitrarily, then NameNode wouldn't be able to distinguish between replicas which do not exist on the DataNode from those that have not been yet reported (see 4 above). Therefore, the proposal is to introduce segmented block reports (SBR), where each report includes a segment of IDs. So the DataNode reports all its replicas in the given range of blockIDs, and if some block is not present in the report, the respective replica must be removed from the NameNode. More details: * NameNode allocates blockIDs sequentially. It should partition the set of allocated so far block IDs into reasonably sized segments. The last segment is open ended. * BlockReportCommand is a new DatanodeCommand, which NameNode should send to a DataNode (in reply to a heartbeat) to order a block report within a specified segment. * When DN receives a BlockReportCommand it forms SBR for the requested segment and sends it to NN. The report also includes the segment boundaries. This could be done per storage. * NN processing of SBR is similar to FBR, but bounded to the reported segment. * NN can eventually start optimizing to request SBRs when it is less busy. * Periodic FBRs, can eventually be removed, but for now should remain for backward compatibility. That is if a DN does not receive any BlockReportCommands from NN, it should send FBR. P.S. There is a lot of jiras discussing partial block reports since prehistoric times. I scanned through many, but found only one mentioning of a similar proposal. In HDFS-395 [~cutting] in [his comment|https://issues.apache.org/jira/browse/HDFS-395?focusedCommentId=12593583=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12593583] posted a link to an old discussion on the topic. Unfortunately the link is now stale. > Segmented Block Reports > --- > > Key: HDFS-11313 > URL: https://issues.apache.org/jira/browse/HDFS-11313 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.2 >Reporter: Konstantin Shvachko > > Block reports from a single DataNode can be currently split into multiple > RPCs each reporting a single DataNode storage (disk). The reports are still > large since disks are getting bigger. Splitting blockReport RPCs into > multiple smaller calls would improve NameNode performance and overall HDFS > stability. > This was discussed in multiple jiras. Here the approach is to let NameNode > divide blockID space into segments and then ask DataNodes to report replicas > in a particular range of IDs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10258) Erasure Coding: support small cluster whose #DataNode < # (Blocks in a BlockGroup)
[ https://issues.apache.org/jira/browse/HDFS-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-10258. Resolution: Later We also committed the XOR 2,1 policy, so I think the priority of this JIRA is lessened. We can revisit if small clusters are found to be important later. > Erasure Coding: support small cluster whose #DataNode < # (Blocks in a > BlockGroup) > -- > > Key: HDFS-10258 > URL: https://issues.apache.org/jira/browse/HDFS-10258 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > > Currently EC has not supported small clusters whose datanode number is > smaller than the block numbers in a block group. This sub task will solve > this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9657) Schedule EC tasks at proper time to reduce the impact of recovery traffic
[ https://issues.apache.org/jira/browse/HDFS-9657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816831#comment-15816831 ] Andrew Wang commented on HDFS-9657: --- Adding the "nice-to-have" label since this seems like a nice JIRA to think about throttling recovery work. > Schedule EC tasks at proper time to reduce the impact of recovery traffic > - > > Key: HDFS-9657 > URL: https://issues.apache.org/jira/browse/HDFS-9657 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-9657-001.patch, HDFS-9657-002.patch > > > The EC recover tasks consume a lot of network bandwidth and disk I/O. > Recovering a corrupt block requires transferring 6 blocks , hence creating a > 6X overhead in network bandwidth and disk I/O. When a datanode fails , the > recovery of the whole blocks on this datanode may use up the network > bandwith. We need to start a recovery task at a proper time in order to give > less impact to the system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9657) Schedule EC tasks at proper time to reduce the impact of recovery traffic
[ https://issues.apache.org/jira/browse/HDFS-9657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-9657: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Schedule EC tasks at proper time to reduce the impact of recovery traffic > - > > Key: HDFS-9657 > URL: https://issues.apache.org/jira/browse/HDFS-9657 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-9657-001.patch, HDFS-9657-002.patch > > > The EC recover tasks consume a lot of network bandwidth and disk I/O. > Recovering a corrupt block requires transferring 6 blocks , hence creating a > 6X overhead in network bandwidth and disk I/O. When a datanode fails , the > recovery of the whole blocks on this datanode may use up the network > bandwith. We need to start a recovery task at a proper time in order to give > less impact to the system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9826) Erasure Coding: Postpone the recovery work for a configurable time period
[ https://issues.apache.org/jira/browse/HDFS-9826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-9826: -- Resolution: Not A Problem Status: Resolved (was: Patch Available) Resolving per above, feel free to reopen if you'd like to resume this work. > Erasure Coding: Postpone the recovery work for a configurable time period > -- > > Key: HDFS-9826 > URL: https://issues.apache.org/jira/browse/HDFS-9826 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Attachments: HDFS-9826-001.patch, HDFS-9826-002.patch > > > Currently NameNode prepares recovering when finding an under replicated > block group. This is inefficient and reduces resources for other operations. > It would be better to postpone the recovery work for a period of time if only > one internal block is corrupted considering points shown by papers such as > \[1\]\[2\]: > 1.Transient errors in which no data are lost account for more than 90% of > data center failures, owing to network partitions, software problems, or > non-disk hardware faults. > 2.Although erasure codes tolerate multiple simultaneous failures, single > failures represent 99.75% of recoveries. > Different clusters may have different status, so we should allow user to > configure the time for postponing the recoveries. Proper configuration will > reduce a large proportion of unnecessary recoveries. When finding multiple > internal blocks corrupted in a block group, we prepare the recovery work > immediately because it’s very rare and we don’t want to increase the risk of > losing data. > [1] Availability in globally distributed storage systems > http://static.usenix.org/events/osdi10/tech/full_papers/Ford.pdf > [2] Rethinking erasure codes for cloud file systems: minimizing I/O for > recovery and degraded reads > http://static.usenix.org/events/fast/tech/full_papers/Khan.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11209) SNN can't checkpoint when rolling upgrade is not finalized
[
https://issues.apache.org/jira/browse/HDFS-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816828#comment-15816828
]
Hadoop QA commented on HDFS-11209:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11209 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846710/HDFS-11209.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux dc6f9d09920a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e692316 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18135/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18135/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18135/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> SNN can't checkpoint when rolling upgrade is not finalized
>
[jira] [Commented] (HDFS-9826) Erasure Coding: Postpone the recovery work for a configurable time period
[ https://issues.apache.org/jira/browse/HDFS-9826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816825#comment-15816825 ] Andrew Wang commented on HDFS-9826: --- As Walter says above, we already prioritize recovery based on the risk of losing data. I'd prefer that we work on improving the throttling mechanisms for recovery work (if necessary) rather than adding a new postpone mechanism. I think we can close this one and keep the more general HDFS-9657 for discussion. > Erasure Coding: Postpone the recovery work for a configurable time period > -- > > Key: HDFS-9826 > URL: https://issues.apache.org/jira/browse/HDFS-9826 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Attachments: HDFS-9826-001.patch, HDFS-9826-002.patch > > > Currently NameNode prepares recovering when finding an under replicated > block group. This is inefficient and reduces resources for other operations. > It would be better to postpone the recovery work for a period of time if only > one internal block is corrupted considering points shown by papers such as > \[1\]\[2\]: > 1.Transient errors in which no data are lost account for more than 90% of > data center failures, owing to network partitions, software problems, or > non-disk hardware faults. > 2.Although erasure codes tolerate multiple simultaneous failures, single > failures represent 99.75% of recoveries. > Different clusters may have different status, so we should allow user to > configure the time for postponing the recoveries. Proper configuration will > reduce a large proportion of unnecessary recoveries. When finding multiple > internal blocks corrupted in a block group, we prepare the recovery work > immediately because it’s very rare and we don’t want to increase the risk of > losing data. > [1] Availability in globally distributed storage systems > http://static.usenix.org/events/osdi10/tech/full_papers/Ford.pdf > [2] Rethinking erasure codes for cloud file systems: minimizing I/O for > recovery and degraded reads > http://static.usenix.org/events/fast/tech/full_papers/Khan.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[
https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816821#comment-15816821
]
Yuanbo Liu commented on HDFS-11150:
---
[~umamaheswararao] Thanks for your response.
{code}
When we finish the movement successfully, we should clean Xattrs.
When we disable SPS dynamically, we should clean Xattrs
{code}
I was considering raising a JIRA to fix it because it's part of retry mechanism
and it's not very easy to remove xattar in
{{BlockStorageMovementAttemptedItems}} right now.
The others look good to me, I'll address them today. Thanks again for your
comments.
> [SPS]: Provide persistence when satisfying storage policy.
> --
>
> Key: HDFS-11150
> URL: https://issues.apache.org/jira/browse/HDFS-11150
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Attachments: HDFS-11150-HDFS-10285.001.patch,
> HDFS-11150-HDFS-10285.002.patch, HDFS-11150-HDFS-10285.003.patch,
> HDFS-11150-HDFS-10285.004.patch, HDFS-11150-HDFS-10285.005.patch,
> HDFS-11150-HDFS-10285.006.patch, editsStored, editsStored.xml
>
>
> Provide persistence for SPS in case that Hadoop cluster crashes by accident.
> Basically we need to change EditLog and FsImage here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size
[
https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816813#comment-15816813
]
Andrew Wang commented on HDFS-9342:
---
This one looks important for correctness. Could one of the watchers comment if
so?
> Erasure coding: client should update and commit block based on acknowledged
> size
>
>
> Key: HDFS-9342
> URL: https://issues.apache.org/jira/browse/HDFS-9342
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Zhe Zhang
>Assignee: Walter Su
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9342.01.patch, HDFS-9342.02.patch,
> HDFS-9342.03.patch
>
>
> For non-EC files, we have:
> {code}
> protected ExtendedBlock block; // its length is number of bytes acked
> {code}
> For EC files, the size of {{DFSStripedOutputStream#currentBlockGroup}} is
> incremented in {{writeChunk}} without waiting for ack. And both
> {{updatePipeline}} and {{commitBlock}} are based on size of
> {{currentBlockGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11313) Segmented Block Reports
Konstantin Shvachko created HDFS-11313: -- Summary: Segmented Block Reports Key: HDFS-11313 URL: https://issues.apache.org/jira/browse/HDFS-11313 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.2 Reporter: Konstantin Shvachko Block reports from a single DataNode can be currently split into multiple RPCs each reporting a single DataNode storage (disk). The reports are still large since disks are getting bigger. Splitting blockReport RPCs into multiple smaller calls would improve NameNode performance and overall HDFS stability. This was discussed in multiple jiras. Here the approach is to let NameNode divide blockID space into segments and then ask DataNodes to report replicas in a particular range of IDs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size
[
https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9342:
--
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Erasure coding: client should update and commit block based on acknowledged
> size
>
>
> Key: HDFS-9342
> URL: https://issues.apache.org/jira/browse/HDFS-9342
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Zhe Zhang
>Assignee: Walter Su
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9342.01.patch, HDFS-9342.02.patch,
> HDFS-9342.03.patch
>
>
> For non-EC files, we have:
> {code}
> protected ExtendedBlock block; // its length is number of bytes acked
> {code}
> For EC files, the size of {{DFSStripedOutputStream#currentBlockGroup}} is
> incremented in {{writeChunk}} without waiting for ack. And both
> {{updatePipeline}} and {{commitBlock}} are based on size of
> {{currentBlockGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10733) NameNode terminated after full GC thinking QJM is unresponsive.
[
https://issues.apache.org/jira/browse/HDFS-10733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinitha Reddy Gankidi updated HDFS-10733:
-
Status: Patch Available (was: Open)
> NameNode terminated after full GC thinking QJM is unresponsive.
> ---
>
> Key: HDFS-10733
> URL: https://issues.apache.org/jira/browse/HDFS-10733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.6.4
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HDFS-10733.001.patch, HDFS-10733.002.patch
>
>
> NameNode went into full GC while in {{AsyncLoggerSet.waitForWriteQuorum()}}.
> After completing GC it checks if the timeout for quorum is reached. If the GC
> was long enough the timeout can expire, and {{QuorumCall.waitFor()}} will
> throw {{TimeoutExcpetion}}. Finally {{FSEditLog.logSync()}} catches the
> exception and terminates NameNode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816802#comment-15816802
]
Andrew Wang commented on HDFS-8095:
---
Got it, thanks for the context Kai. When you say data corruption, do you mean
data loss if the cluster has insufficient # datanodes or racks?
Overall I like the idea of making this act like the blocksize and replication
factor. These are specified by the client and have default values in client
configuration, but the NN enforces limits on the values to make sure they're
reasonable.
Seems related to HDFS-7859. Admins would add all the allowable EC policies, and
users would can choose among them. NN would throw an error if an unknown policy
is requested.
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10999) Use more generic "low redundancy" blocks instead of "under replicated" blocks
[ https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816787#comment-15816787 ] Yuanbo Liu commented on HDFS-10999: --- I think I've lost the context of this JIRA. Dismiss my ownership, feel free to assign it to anyone else. Sorry to interrupt! > Use more generic "low redundancy" blocks instead of "under replicated" blocks > - > > Key: HDFS-10999 > URL: https://issues.apache.org/jira/browse/HDFS-10999 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Yuanbo Liu > Labels: hdfs-ec-3.0-nice-to-have, supportability > > Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic > term "low redundancy" to the old-fashioned "under replicated". But this term > is still being used in messages in several places, such as web ui, dfsadmin > and fsck. We should probably change them to avoid confusion. > File this jira to discuss it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10999) Use more generic "low redundancy" blocks instead of "under replicated" blocks
[ https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-10999: -- Assignee: (was: Yuanbo Liu) > Use more generic "low redundancy" blocks instead of "under replicated" blocks > - > > Key: HDFS-10999 > URL: https://issues.apache.org/jira/browse/HDFS-10999 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang > Labels: hdfs-ec-3.0-nice-to-have, supportability > > Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic > term "low redundancy" to the old-fashioned "under replicated". But this term > is still being used in messages in several places, such as web ui, dfsadmin > and fsck. We should probably change them to avoid confusion. > File this jira to discuss it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11268) Persist erasure coding policy ID in FSImage
[ https://issues.apache.org/jira/browse/HDFS-11268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816774#comment-15816774 ] SammiChen commented on HDFS-11268: -- Hi [~andrew.wang], I'am working on it. Will give an update soon. > Persist erasure coding policy ID in FSImage > --- > > Key: HDFS-11268 > URL: https://issues.apache.org/jira/browse/HDFS-11268 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: SammiChen >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > Currently, FSImage only has the information about whether the file is striped > or not. It doesn't save the erasure coding policy ID. Later, when the FSImage > is loaded to create the name space, the default system ec policy is used to > as file's ec policy. In case if the ec policy on file is not the default ec > policy, then the content of the file cannot be accessed correctly in this > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11306) Print remaining edit logs from buffer if edit log can't be rolled.
[ https://issues.apache.org/jira/browse/HDFS-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11306: --- Attachment: HDFS-11306.002.patch Upload v002 patch to address [~yzhangal]'s comment. > Print remaining edit logs from buffer if edit log can't be rolled. > -- > > Key: HDFS-11306 > URL: https://issues.apache.org/jira/browse/HDFS-11306 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HDFS-11306.001.patch, HDFS-11306.002.patch > > > In HDFS-10943 [~yzhangal] reported that edit log can not be rolled due to > unexpected edit logs lingering in the buffer. > Unable to root cause the bug, I propose that we dump the remaining edit logs > in the buffer into namenode log, before crashing namenode. Use this new > capability to find the ops that sneaks into the buffer unexpectedly, and > hopefully catch the bug. > This effort is orthogonal, but related to HDFS-11292, which adds additional > informational logs to help debug this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816745#comment-15816745
]
Hadoop QA commented on HDFS-8201:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-8201 does not apply to HDFS-7285. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8201 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12734445/HDFS-8201-HDFS-7285.005.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18138/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor the end to end test for stripping file writing and reading
> ---
>
> Key: HDFS-8201
> URL: https://issues.apache.org/jira/browse/HDFS-8201
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Xinwei Qin
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8201-HDFS-7285.003.patch,
> HDFS-8201-HDFS-7285.004.patch, HDFS-8201-HDFS-7285.005.patch,
> HDFS-8201.001.patch, HDFS-8201.002.patch
>
>
> According to off-line discussion with [~zhz] and [~xinwei], we need to
> implement an end to end test for stripping file support:
> * Create an EC zone;
> * Create a file in the zone;
> * Write various typical sizes of content to the file, each size maybe a test
> method;
> * Read the written content back;
> * Compare the written content and read content to ensure it's good;
> This jira aims to refactor the end to end test
> class(TestWriteReadStripedFile) in order to reuse them conveniently in the
> next test step for erasure encoding and recovering. Will open separate issue
> for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816734#comment-15816734
]
Kai Zheng commented on HDFS-8095:
-
Thanks [~andrew.wang] for the thoughts. I'm not sure if we still need this. In
the current implementation we do allow users to specify a EC policy for a
folder/file but if otherwise they don't provide, a hard-coded default one is
used. Allowing to configure/change the system default policy may be error-prone
because if the configuration is used improperly, it may cause data corruption.
Note the default policy is {{RS-6-3-64K}} and the referenced schema {{RS-6-3}}
is widely adopted in the industry, so the default policy should be good enough
considering the default cell size of {{64k}} should be a good choice (well
discussed but not sure where now).
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-8796) Erasure coding: merge HDFS-8499 to EC branch and refactor BlockInfoStriped
[ https://issues.apache.org/jira/browse/HDFS-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-8796. --- Resolution: Invalid I think this JIRA is invalid now given that the EC branch has been merged to trunk, resolving. > Erasure coding: merge HDFS-8499 to EC branch and refactor BlockInfoStriped > -- > > Key: HDFS-8796 > URL: https://issues.apache.org/jira/browse/HDFS-8796 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8796-HDFS-7285.00.patch, > HDFS-8796-HDFS-7285.01-part1.patch, HDFS-8796-HDFS-7285.01-part2.patch > > > Separating this change from the HDFS-8728 discussion. Per suggestion from > [~szetszwo], clarifying the description of the change. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8387) Erasure Coding: Revisit the long and int datatypes usage in striping logic
[
https://issues.apache.org/jira/browse/HDFS-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816731#comment-15816731
]
Andrew Wang commented on HDFS-8387:
---
Hi [~rakeshr] do you think this JIRA is still important, and have plans to work
on it?
> Erasure Coding: Revisit the long and int datatypes usage in striping logic
> --
>
> Key: HDFS-8387
> URL: https://issues.apache.org/jira/browse/HDFS-8387
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Labels: hdfs-ec-3.0-nice-to-have
>
> This idea of this jira is to revisit the usage of {{long}} and {{int}} data
> types in the striping logic.
> Related discussion
> [here|https://issues.apache.org/jira/browse/HDFS-8294?focusedCommentId=14540788=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14540788]
> in HDFS-8294
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8387) Erasure Coding: Revisit the long and int datatypes usage in striping logic
[
https://issues.apache.org/jira/browse/HDFS-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8387:
--
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Erasure Coding: Revisit the long and int datatypes usage in striping logic
> --
>
> Key: HDFS-8387
> URL: https://issues.apache.org/jira/browse/HDFS-8387
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Labels: hdfs-ec-3.0-nice-to-have
>
> This idea of this jira is to revisit the usage of {{long}} and {{int}} data
> types in the striping logic.
> Related discussion
> [here|https://issues.apache.org/jira/browse/HDFS-8294?focusedCommentId=14540788=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14540788]
> in HDFS-8294
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9345) Erasure Coding: create dummy coder and schema
[ https://issues.apache.org/jira/browse/HDFS-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-9345: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Erasure Coding: create dummy coder and schema > - > > Key: HDFS-9345 > URL: https://issues.apache.org/jira/browse/HDFS-9345 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Labels: hdfs-ec-3.0-nice-to-have > > We can create dummy coder which does no computation and simply returns zero > bytes. Similarly, we can create a test-only schema with no parity blocks. > Such coder and schema can be used to isolate the performance issue to > HDFS-side logic instead of codec, which would be useful when tuning > performance of EC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8201: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Refactor the end to end test for stripping file writing and reading > --- > > Key: HDFS-8201 > URL: https://issues.apache.org/jira/browse/HDFS-8201 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Xinwei Qin > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8201-HDFS-7285.003.patch, > HDFS-8201-HDFS-7285.004.patch, HDFS-8201-HDFS-7285.005.patch, > HDFS-8201.001.patch, HDFS-8201.002.patch > > > According to off-line discussion with [~zhz] and [~xinwei], we need to > implement an end to end test for stripping file support: > * Create an EC zone; > * Create a file in the zone; > * Write various typical sizes of content to the file, each size maybe a test > method; > * Read the written content back; > * Compare the written content and read content to ensure it's good; > This jira aims to refactor the end to end test > class(TestWriteReadStripedFile) in order to reuse them conveniently in the > next test step for erasure encoding and recovering. Will open separate issue > for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8201: -- Target Version/s: 3.0.0-alpha2 (was: HDFS-7285) > Refactor the end to end test for stripping file writing and reading > --- > > Key: HDFS-8201 > URL: https://issues.apache.org/jira/browse/HDFS-8201 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Xinwei Qin > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8201-HDFS-7285.003.patch, > HDFS-8201-HDFS-7285.004.patch, HDFS-8201-HDFS-7285.005.patch, > HDFS-8201.001.patch, HDFS-8201.002.patch > > > According to off-line discussion with [~zhz] and [~xinwei], we need to > implement an end to end test for stripping file support: > * Create an EC zone; > * Create a file in the zone; > * Write various typical sizes of content to the file, each size maybe a test > method; > * Read the written content back; > * Compare the written content and read content to ensure it's good; > This jira aims to refactor the end to end test > class(TestWriteReadStripedFile) in order to reuse them conveniently in the > next test step for erasure encoding and recovering. Will open separate issue > for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8225) EC client code should not print info log message
[ https://issues.apache.org/jira/browse/HDFS-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8225: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > EC client code should not print info log message > > > Key: HDFS-8225 > URL: https://issues.apache.org/jira/browse/HDFS-8225 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Labels: hdfs-ec-3.0-nice-to-have > > There are many LOG.info(..) calls in the code. We should either remove them > or change the log level. Users don't want to see any log message on the > screen when running the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11302) Improve Logging for SSLHostnameVerifier
[
https://issues.apache.org/jira/browse/HDFS-11302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816714#comment-15816714
]
Chen Liang commented on HDFS-11302:
---
The v001 patch is following the existing style of this class, which is
different from Jenkins desired style. This is the cause of all the checkstyle
complains here.
> Improve Logging for SSLHostnameVerifier
> ---
>
> Key: HDFS-11302
> URL: https://issues.apache.org/jira/browse/HDFS-11302
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-11302.001.patch
>
>
> SSLHostnameVerifier interface/class was copied from other projects without
> any logging to help troubleshooting SSL certificate related issues. For a
> misconfigured SSL truststore, we may get some very confusing error message
> like
> {code}
> >hdfs dfs -cat swebhdfs://NNl/tmp/test1.txt
> ...
> cause:java.io.IOException: DN2:50475: HTTPS hostname wrong: should be
> cat: DN2:50475: HTTPS hostname wrong: should be
> {code}
> This ticket is opened to add tracing to give more useful context information
> around SSL certificate verification failures inside the following code.
> {code}AbstractVerifier#check(String[] host, X509Certificate cert) {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10733) NameNode terminated after full GC thinking QJM is unresponsive.
[
https://issues.apache.org/jira/browse/HDFS-10733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinitha Reddy Gankidi updated HDFS-10733:
-
Attachment: HDFS-10733.002.patch
> NameNode terminated after full GC thinking QJM is unresponsive.
> ---
>
> Key: HDFS-10733
> URL: https://issues.apache.org/jira/browse/HDFS-10733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.6.4
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HDFS-10733.001.patch, HDFS-10733.002.patch
>
>
> NameNode went into full GC while in {{AsyncLoggerSet.waitForWriteQuorum()}}.
> After completing GC it checks if the timeout for quorum is reached. If the GC
> was long enough the timeout can expire, and {{QuorumCall.waitFor()}} will
> throw {{TimeoutExcpetion}}. Finally {{FSEditLog.logSync()}} catches the
> exception and terminates NameNode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10733) NameNode terminated after full GC thinking QJM is unresponsive.
[
https://issues.apache.org/jira/browse/HDFS-10733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816705#comment-15816705
]
Vinitha Reddy Gankidi commented on HDFS-10733:
--
[~shv] I agree. Attached a new patch with this change.
> NameNode terminated after full GC thinking QJM is unresponsive.
> ---
>
> Key: HDFS-10733
> URL: https://issues.apache.org/jira/browse/HDFS-10733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.6.4
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HDFS-10733.001.patch, HDFS-10733.002.patch
>
>
> NameNode went into full GC while in {{AsyncLoggerSet.waitForWriteQuorum()}}.
> After completing GC it checks if the timeout for quorum is reached. If the GC
> was long enough the timeout can expire, and {{QuorumCall.waitFor()}} will
> throw {{TimeoutExcpetion}}. Finally {{FSEditLog.logSync()}} catches the
> exception and terminates NameNode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8295) Add MODIFY and REMOVE ECSchema editlog operations
[ https://issues.apache.org/jira/browse/HDFS-8295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8295: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Add MODIFY and REMOVE ECSchema editlog operations > - > > Key: HDFS-8295 > URL: https://issues.apache.org/jira/browse/HDFS-8295 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xinwei Qin >Assignee: Xinwei Qin > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8295.001.patch > > > If MODIFY and REMOVE ECSchema operations are supported, then add these > editlog operations to persist them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816693#comment-15816693
]
Andrew Wang commented on HDFS-8095:
---
Adding the "nice-to-have" label, since I think it's worth having a discussion
about the behavior of the "default policy" and whether we need one.
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8196: -- Labels: NameNode WebUI hdfs-ec-3.0-nice-to-have (was: NameNode WebUI) > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11296) Maintenance state expiry should be an epoch time and not jvm monotonic
[
https://issues.apache.org/jira/browse/HDFS-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11296:
--
Attachment: HDFS-11296-branch-2.01.patch
> Maintenance state expiry should be an epoch time and not jvm monotonic
> --
>
> Key: HDFS-11296
> URL: https://issues.apache.org/jira/browse/HDFS-11296
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11296-branch-2.01.patch, HDFS-11296.01.patch
>
>
> Currently it is possible to configure an expiry time in milliseconds for a
> DataNode in maintenance state. As per the design, the expiry attribute is an
> absolute time, beyond which NameNode starts to stop the ongoing maintenance
> operation for that DataNode. Internally in the code, this expiry time is read
> and checked against {{Time.monotonicNow()}} making the expiry based on more
> of JVM's runtime, which is very difficult to configure for any external user.
> The goal is to make the expiry time an absolute epoch time, so that its easy
> to configure for external users.
> {noformat}
> {
> "hostName": ,
> "port": ,
> "adminState": "IN_MAINTENANCE",
> "maintenanceExpireTimeInMS":
> }
> {noformat}
> DatanodeInfo.java
> {noformat}
> public static boolean maintenanceNotExpired(long maintenanceExpireTimeInMS)
> {
> return Time.monotonicNow() < maintenanceExpireTimeInMS;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816691#comment-15816691
]
Andrew Wang commented on HDFS-8095:
---
Is this JIRA superceded by HDFS-7859? Also like [Vinay's
comment|https://issues.apache.org/jira/browse/HDFS-8074?focusedCommentId=14484952=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14484952]
on HDFS-8074, I wonder why we can't just require that the admin specify the
policy when creating the EC zone. Alternatively, it could be a default in the
client config like the replication factor.
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8095) Allow to configure the system default EC schema
[
https://issues.apache.org/jira/browse/HDFS-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8095:
--
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Allow to configure the system default EC schema
> ---
>
> Key: HDFS-8095
> URL: https://issues.apache.org/jira/browse/HDFS-8095
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
>
> As suggested by [~umamaheswararao] and [~vinayrpet] in HDFS-8074, we may
> desire allowing to configure the system default EC schema, so in any
> deployment a cluster admin may be able to define their own system default
> one. In the discussion, we have two approaches to configure the system
> default schema: 1) predefine it in the {{ecschema-def.xml}} file, making sure
> it's not changed; 2) configure the key parameter values as properties in
> {{core-site.xml}}. Open this for future consideration in case it's forgotten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8140) ECSchema supports for offline EditsVistor over an OEV XML file
[ https://issues.apache.org/jira/browse/HDFS-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8140: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > ECSchema supports for offline EditsVistor over an OEV XML file > -- > > Key: HDFS-8140 > URL: https://issues.apache.org/jira/browse/HDFS-8140 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Xinwei Qin >Assignee: Xinwei Qin > Labels: hdfs-ec-3.0-nice-to-have > > Make the ECSchema info in Editlog Support for offline EditsVistor over an OEV > XML file, which is not implemented in HDFS-7859. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8140) ECSchema supports for offline EditsVisitor over an OEV XML file
[ https://issues.apache.org/jira/browse/HDFS-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8140: -- Summary: ECSchema supports for offline EditsVisitor over an OEV XML file (was: ECSchema supports for offline EditsVistor over an OEV XML file) > ECSchema supports for offline EditsVisitor over an OEV XML file > --- > > Key: HDFS-8140 > URL: https://issues.apache.org/jira/browse/HDFS-8140 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Xinwei Qin >Assignee: Xinwei Qin > Labels: hdfs-ec-3.0-nice-to-have > > Make the ECSchema info in Editlog Support for offline EditsVistor over an OEV > XML file, which is not implemented in HDFS-7859. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8140) ECSchema supports for offline EditsVistor over an OEV XML file
[ https://issues.apache.org/jira/browse/HDFS-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8140: -- Target Version/s: 3.0.0-alpha2 (was: HDFS-7285) > ECSchema supports for offline EditsVistor over an OEV XML file > -- > > Key: HDFS-8140 > URL: https://issues.apache.org/jira/browse/HDFS-8140 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Xinwei Qin >Assignee: Xinwei Qin > Labels: hdfs-ec-3.0-nice-to-have > > Make the ECSchema info in Editlog Support for offline EditsVistor over an OEV > XML file, which is not implemented in HDFS-7859. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7619) Erasure Coding: Handling truncated files in snapshots
[ https://issues.apache.org/jira/browse/HDFS-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7619: -- Summary: Erasure Coding: Handling truncated files in snapshots (was: Erasure Coding: Handling files in snapshots ) > Erasure Coding: Handling truncated files in snapshots > -- > > Key: HDFS-7619 > URL: https://issues.apache.org/jira/browse/HDFS-7619 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Takuya Fukudome > > Currently for files within snapshots, the file diff only records block list > (before the truncate feature we only recorded file length). Now the file diff > should also be able to capture the block groups. > In the meanwhile, if a file has been truncated in history and the change has > been recorded in snapshots, when we convert the file to EC, the conversion > should also cover the data that were truncated and the corresponding file > diff needs to be updated accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7786) Handle slow writers for DFSStripedOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7786: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Handle slow writers for DFSStripedOutputStream > -- > > Key: HDFS-7786 > URL: https://issues.apache.org/jira/browse/HDFS-7786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Keisuke Ogiwara > Labels: hdfs-ec-3.0-nice-to-have > Fix For: HDFS-7285 > > > The streamers in DFSStripedOutputStream may have different write speed. We > need to consider and handle the situation when one or more streamers begin to > write slowly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9391) Update webUI/JMX to display maintenance state info
[ https://issues.apache.org/jira/browse/HDFS-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy updated HDFS-9391: - Attachment: HDFS-9391-branch-2.02.patch HDFS-9391-branch-2-MaintenanceMode-WebUI.pdf Thanks [~mingma] for the review. Attached branch2.02 patch fixing the LeavingServiceStatus function arguments and WebUI samples for branch-2. > Update webUI/JMX to display maintenance state info > -- > > Key: HDFS-9391 > URL: https://issues.apache.org/jira/browse/HDFS-9391 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Ming Ma >Assignee: Manoj Govindassamy > Attachments: HDFS-9391-MaintenanceMode-WebUI.pdf, > HDFS-9391-branch-2-MaintenanceMode-WebUI.pdf, HDFS-9391-branch-2.01.patch, > HDFS-9391-branch-2.02.patch, HDFS-9391.01.patch, HDFS-9391.02.patch, > HDFS-9391.03.patch, HDFS-9391.04.patch, Maintenance webUI.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7337: -- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Configurable and pluggable Erasure Codec and schema > --- > > Key: HDFS-7337 > URL: https://issues.apache.org/jira/browse/HDFS-7337 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Kai Zheng > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-7337-prototype-v1.patch, > HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip, > PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, > PluggableErasureCodec.pdf > > > According to HDFS-7285 and the design, this considers to support multiple > Erasure Codecs via pluggable approach. It allows to define and configure > multiple codec schemas with different coding algorithms and parameters. The > resultant codec schemas can be utilized and specified via command tool for > different file folders. While design and implement such pluggable framework, > it’s also to implement a concrete codec by default (Reed Solomon) to prove > the framework is useful and workable. Separate JIRA could be opened for the > RS codec implementation. > Note HDFS-7353 will focus on the very low level codec API and implementation > to make concrete vendor libraries transparent to the upper layer. This JIRA > focuses on high level stuffs that interact with configuration, schema and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7337: -- Target Version/s: 3.0.0-alpha2 (was: HDFS-7285) > Configurable and pluggable Erasure Codec and schema > --- > > Key: HDFS-7337 > URL: https://issues.apache.org/jira/browse/HDFS-7337 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Kai Zheng > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-7337-prototype-v1.patch, > HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip, > PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, > PluggableErasureCodec.pdf > > > According to HDFS-7285 and the design, this considers to support multiple > Erasure Codecs via pluggable approach. It allows to define and configure > multiple codec schemas with different coding algorithms and parameters. The > resultant codec schemas can be utilized and specified via command tool for > different file folders. While design and implement such pluggable framework, > it’s also to implement a concrete codec by default (Reed Solomon) to prove > the framework is useful and workable. Separate JIRA could be opened for the > RS codec implementation. > Note HDFS-7353 will focus on the very low level codec API and implementation > to make concrete vendor libraries transparent to the upper layer. This JIRA > focuses on high level stuffs that interact with configuration, schema and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11302) Improve Logging for SSLHostnameVerifier
[
https://issues.apache.org/jira/browse/HDFS-11302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1581#comment-1581
]
Hadoop QA commented on HDFS-11302:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 10 new + 318 unchanged - 1 fixed = 328 total (was 319) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
25s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11302 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846694/HDFS-11302.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d5a4be425ac7 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e692316 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18134/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18134/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18134/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve Logging for SSLHostnameVerifier
> ---
>
> Key: HDFS-11302
> URL: https://issues.apache.org/jira/browse/HDFS-11302
> Project: Hadoop HDFS
> Issue Type:
[jira] [Commented] (HDFS-11028) libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel pending connections
[
https://issues.apache.org/jira/browse/HDFS-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816662#comment-15816662
]
Hadoop QA commented on HDFS-11028:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
59s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
31s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
54s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
52s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with
JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:78fc6b6 |
| JIRA Issue | HDFS-11028 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846699/HDFS-11028.HDFS-8707.003.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux ec758400ee90 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / 02cef3e |
| Default Java | 1.7.0_121 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_111
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 |
| JDK v1.7.0_121 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18133/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18133/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> libhdfs++: FileHandleImpl::CancelOperations needs to be able to cancel
> pending connections
> --
>
> Key: HDFS-11028
> URL: https://issues.apache.org/jira/browse/HDFS-11028
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James
[jira] [Updated] (HDFS-8672) Erasure Coding: Add EC-related Metrics to NN (seperate striped blocks count from UnderReplicatedBlocks count)
[
https://issues.apache.org/jira/browse/HDFS-8672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8672:
--
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Erasure Coding: Add EC-related Metrics to NN (seperate striped blocks count
> from UnderReplicatedBlocks count)
> -
>
> Key: HDFS-8672
> URL: https://issues.apache.org/jira/browse/HDFS-8672
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Labels: hdfs-ec-3.0-nice-to-have
>
> 1. {{MissingBlocks}} metric is updated in HDFS-8461 so it includes striped
> blocks.
> 2. {{CorruptBlocks}} metric is updated in HDFS-8619 so it includes striped
> blocks.
> 3. {{UnderReplicatedBlocks}} and {{PendingReplicationBlocks}} includes
> striped blocks (HDFS-7912).
> This jira aims to seperate striped blocks count from
> {{UnderReplicatedBlocks}} count.
> EC file recovery need coding. It's more expensive than block duplication.
> It's necessary to seperate striped blocks count from UnderReplicatedBlocks
> count. So user can know what's going on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10999) Use more generic "low redundancy" blocks instead of "under replicated" blocks
[ https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10999: --- Labels: hdfs-ec-3.0-nice-to-have supportability (was: supportability) > Use more generic "low redundancy" blocks instead of "under replicated" blocks > - > > Key: HDFS-10999 > URL: https://issues.apache.org/jira/browse/HDFS-10999 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Yuanbo Liu > Labels: hdfs-ec-3.0-nice-to-have, supportability > > Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic > term "low redundancy" to the old-fashioned "under replicated". But this term > is still being used in messages in several places, such as web ui, dfsadmin > and fsck. We should probably change them to avoid confusion. > File this jira to discuss it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-7674) [umbrella] Adding metrics for Erasure Coding
[ https://issues.apache.org/jira/browse/HDFS-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-7674. --- Resolution: Done I think we can close this since the subtasks are resolved. Thanks everyone for the hard work! > [umbrella] Adding metrics for Erasure Coding > > > Key: HDFS-7674 > URL: https://issues.apache.org/jira/browse/HDFS-7674 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Li Bo > > As the design (in HDFS-7285) indicates, erasure coding involves non-trivial > impact and workload for NameNode, DataNode and client; it also allows > configurable and pluggable erasure codec and schema with flexible tradeoff > options (see HDFS-7337). To support necessary analysis and adjustment, we'd > better have various meaningful metrics for the EC support, like > encoding/decoding tasks, recovered blocks, read/transferred data size, > computation time and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11302) Improve Logging for SSLHostnameVerifier
[
https://issues.apache.org/jira/browse/HDFS-11302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816640#comment-15816640
]
Xiaoyu Yao commented on HDFS-11302:
---
Thanks [~vagarychen] for working on this. Patch looks good to me. +1 pending
Jenkins.
> Improve Logging for SSLHostnameVerifier
> ---
>
> Key: HDFS-11302
> URL: https://issues.apache.org/jira/browse/HDFS-11302
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-11302.001.patch
>
>
> SSLHostnameVerifier interface/class was copied from other projects without
> any logging to help troubleshooting SSL certificate related issues. For a
> misconfigured SSL truststore, we may get some very confusing error message
> like
> {code}
> >hdfs dfs -cat swebhdfs://NNl/tmp/test1.txt
> ...
> cause:java.io.IOException: DN2:50475: HTTPS hostname wrong: should be
> cat: DN2:50475: HTTPS hostname wrong: should be
> {code}
> This ticket is opened to add tracing to give more useful context information
> around SSL certificate verification failures inside the following code.
> {code}AbstractVerifier#check(String[] host, X509Certificate cert) {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816623#comment-15816623 ] Andrew Wang commented on HDFS-11096: Thanks for checking all this Sean, a couple replies: * Nice find on HDFS-11312, particularly since it's also an issue in branch-2.8. * Not sure on the string -> type conversion, but it's in branch-2.8 so it does need confirmation real soon. I don't see a compatibility discussion on YARN-3583 where this was done. * The discussion on YARN-4844 explains that int32 and int64 are both just varints under the hood, so upgrading is okay except for the overflow issue. Since the point of upgrading the types is to fix overflow, this seems okay to me. * Moving messages between files seems fine to me, as long we also updated references to point to the new locations. > Support rolling upgrade between 2.x and 3.x > --- > > Key: HDFS-11096 > URL: https://issues.apache.org/jira/browse/HDFS-11096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rolling upgrades >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > > trunk has a minimum software version of 3.0.0-alpha1. This means we can't > rolling upgrade between branch-2 and trunk. > This is a showstopper for large deployments. Unless there are very compelling > reasons to break compatibility, let's restore the ability to rolling upgrade > to 3.x releases. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11209) SNN can't checkpoint when rolling upgrade is not finalized
[
https://issues.apache.org/jira/browse/HDFS-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11209:
--
Attachment: HDFS-11209.03.patch
Update patch for HDFS-11209.03.patch. Diff from prev: consolidate the NN
layout version update logic into FSImage#updateStorageVersion().
> SNN can't checkpoint when rolling upgrade is not finalized
> --
>
> Key: HDFS-11209
> URL: https://issues.apache.org/jira/browse/HDFS-11209
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Critical
> Attachments: HDFS-11209.00.patch, HDFS-11209.01.patch,
> HDFS-11209.02.patch, HDFS-11209.03.patch
>
>
> Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432
> brings this back.
> With HDFS-8432, the primary NN will not update the VERSION file to the new
> version after running with "rollingUpgrade" option until upgrade is
> finalized. This is to support more downgrade use cases.
> However, the checkpoint on the SNN is incorrectly updating the VERSION file
> when the rollingUpgrade is not finalized yet. As a result, the SNN checkpoint
> successfully but fail to push it to the primary NN because its version is
> higher than the primary NN as shown below.
> {code}
> 2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode
> (SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException:
> Image uploading failed, status: 403, url:
> http://NN:50070/imagetransfer?txid=345404754=IMAGE...,
> message: This namenode has storage info -60:221856466:1444080250181:clusterX
> but the secondary expected -63:221856466:1444080250181:clusterX
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11209) SNN can't checkpoint when rolling upgrade is not finalized
[
https://issues.apache.org/jira/browse/HDFS-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816608#comment-15816608
]
Xiaoyu Yao edited comment on HDFS-11209 at 1/11/17 12:07 AM:
-
Update patch for HDFS-11209.03.patch. Diff from prev: consolidate the NN
layout version update logic into FSImage#updateStorageVersion() to avoid future
bugs in this area.
was (Author: xyao):
Update patch for HDFS-11209.03.patch. Diff from prev: consolidate the NN
layout version update logic into FSImage#updateStorageVersion().
> SNN can't checkpoint when rolling upgrade is not finalized
> --
>
> Key: HDFS-11209
> URL: https://issues.apache.org/jira/browse/HDFS-11209
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Critical
> Attachments: HDFS-11209.00.patch, HDFS-11209.01.patch,
> HDFS-11209.02.patch, HDFS-11209.03.patch
>
>
> Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432
> brings this back.
> With HDFS-8432, the primary NN will not update the VERSION file to the new
> version after running with "rollingUpgrade" option until upgrade is
> finalized. This is to support more downgrade use cases.
> However, the checkpoint on the SNN is incorrectly updating the VERSION file
> when the rollingUpgrade is not finalized yet. As a result, the SNN checkpoint
> successfully but fail to push it to the primary NN because its version is
> higher than the primary NN as shown below.
> {code}
> 2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode
> (SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException:
> Image uploading failed, status: 403, url:
> http://NN:50070/imagetransfer?txid=345404754=IMAGE...,
> message: This namenode has storage info -60:221856466:1444080250181:clusterX
> but the secondary expected -63:221856466:1444080250181:clusterX
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11303) Hedged read might hang infinitely if read data from all DN failed
[
https://issues.apache.org/jira/browse/HDFS-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816550#comment-15816550
]
Hadoop QA commented on HDFS-11303:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
57 unchanged - 0 fixed = 58 total (was 57) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11303 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846229/HDFS-11303-001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 46ed72ff709f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e692316 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18132/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/18132/artifact/patchprocess/whitespace-eol.txt
|
| unit |
[jira] [Commented] (HDFS-11312) Discrepancy in nonDfsUsed index in protobuf
[ https://issues.apache.org/jira/browse/HDFS-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816546#comment-15816546 ] Andrew Wang commented on HDFS-11312: It looks like this discrepancy exists between 2.8 and 2.7, so marking this as a blocker for 2.8.0 as well. [~arpitagarwal] / [~brahmareddy] could you take a look at Sean's patch since you have context from HDFS-9038? > Discrepancy in nonDfsUsed index in protobuf > --- > > Key: HDFS-11312 > URL: https://issues.apache.org/jira/browse/HDFS-11312 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Blocker > Attachments: HDFS-11312.001.patch > > > The patches for HDFS-9038 had a discrepancy between trunk and branch-2.7: in > one message type, nonDfsUsed is given 2 different indices. This is a minor > wire incompatibility that is easy to fix... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11312) Discrepancy in nonDfsUsed index in protobuf
[ https://issues.apache.org/jira/browse/HDFS-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11312: --- Affects Version/s: 2.8.0 3.0.0-alpha1 Target Version/s: 2.8.0, 3.0.0-alpha2 Priority: Blocker (was: Minor) > Discrepancy in nonDfsUsed index in protobuf > --- > > Key: HDFS-11312 > URL: https://issues.apache.org/jira/browse/HDFS-11312 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Blocker > Attachments: HDFS-11312.001.patch > > > The patches for HDFS-9038 had a discrepancy between trunk and branch-2.7: in > one message type, nonDfsUsed is given 2 different indices. This is a minor > wire incompatibility that is easy to fix... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[
https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816514#comment-15816514
]
Hadoop QA commented on HDFS-10702:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 47s{color} | {color:orange} root: The patch generated 10 new + 1003
unchanged - 2 fixed = 1013 total (was 1005) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
7s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestStaleReadFs |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10702 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846654/HDFS-10702.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux bf25b022a3d5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e692316 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0
[jira] [Commented] (HDFS-11150) [SPS]: Provide persistence when satisfying storage policy.
[
https://issues.apache.org/jira/browse/HDFS-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816508#comment-15816508
]
Uma Maheswara Rao G commented on HDFS-11150:
Latest patch almost looks good to me. A few minor improvements on tests
# .
{quote}
+// // test directory
{quote}
Can you remove this double comment?
# .
{code}
@Test
{code}
Please add timeouts for tests
Also add java doc for each test what they are doing.
# .
{code}
"WARM"
{code}
Can we make these string as constants in the file?
# .
typo: DataNodes' —> DataNode’s
# .
I think waitExpectedStorageType is duplicate from other testfiles. Can we move
this to DFSTestUtil.java and rename method to waitForExpectedStorageType
I think
TestStoragePolicySatisfyWorker#waitForLocatedBlockWithArchiveStorageType also
can use this common method.
At last, tasks to address are:
# When we finish the movement successfully, we should clean Xattrs.
# When we disable SPS dynamically, we should clean Xattrs
I am ok to handle them in separate JIRA. Could you be able to raise a JIRA to
track them? (They both can be handled in single JIRA IMO)
> [SPS]: Provide persistence when satisfying storage policy.
> --
>
> Key: HDFS-11150
> URL: https://issues.apache.org/jira/browse/HDFS-11150
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Attachments: HDFS-11150-HDFS-10285.001.patch,
> HDFS-11150-HDFS-10285.002.patch, HDFS-11150-HDFS-10285.003.patch,
> HDFS-11150-HDFS-10285.004.patch, HDFS-11150-HDFS-10285.005.patch,
> HDFS-11150-HDFS-10285.006.patch, editsStored, editsStored.xml
>
>
> Provide persistence for SPS in case that Hadoop cluster crashes by accident.
> Basically we need to change EditLog and FsImage here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816505#comment-15816505 ] Sean Mackrory commented on HDFS-11096: -- The getHdfsBlockLocations removal is documented in HDFS-8895 - apparently that had been deprecated. I filed HDFS-11312 and posted a patch for the nonDfsUsed discrepancy. > Support rolling upgrade between 2.x and 3.x > --- > > Key: HDFS-11096 > URL: https://issues.apache.org/jira/browse/HDFS-11096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rolling upgrades >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > > trunk has a minimum software version of 3.0.0-alpha1. This means we can't > rolling upgrade between branch-2 and trunk. > This is a showstopper for large deployments. Unless there are very compelling > reasons to break compatibility, let's restore the ability to rolling upgrade > to 3.x releases. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11312) Discrepancy in nonDfsUsed index in protobuf
[ https://issues.apache.org/jira/browse/HDFS-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HDFS-11312: - Attachment: HDFS-11312.001.patch > Discrepancy in nonDfsUsed index in protobuf > --- > > Key: HDFS-11312 > URL: https://issues.apache.org/jira/browse/HDFS-11312 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Attachments: HDFS-11312.001.patch > > > The patches for HDFS-9038 had a discrepancy between trunk and branch-2.7: in > one message type, nonDfsUsed is given 2 different indices. This is a minor > wire incompatibility that is easy to fix... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11312) Discrepancy in nonDfsUsed index in protobuf
Sean Mackrory created HDFS-11312: Summary: Discrepancy in nonDfsUsed index in protobuf Key: HDFS-11312 URL: https://issues.apache.org/jira/browse/HDFS-11312 Project: Hadoop HDFS Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory Priority: Minor The patches for HDFS-9038 had a discrepancy between trunk and branch-2.7: in one message type, nonDfsUsed is given 2 different indices. This is a minor wire incompatibility that is easy to fix... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
