[jira] [Commented] (HDFS-11634) Optimize BlockIterator when interating starts in the middle.
[
https://issues.apache.org/jira/browse/HDFS-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961669#comment-15961669
]
Hadoop QA commented on HDFS-11634:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 147 unchanged - 0 fixed = 148 total (was 147) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11634 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862554/HDFS-11634.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8f18acb67771 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2aa8967 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19021/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19021/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19021/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19021/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[
https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961667#comment-15961667
]
Hadoop QA commented on HDFS-11384:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 251 unchanged - 0 fixed = 252 total (was 251) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11384 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862555/HDFS-11384.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 70b6452eb18d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2aa8967 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19022/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19022/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19022/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19022/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add option for balancer to disperse getBlocks calls to avoid NameNode's
> rpc.CallQueueLength spike
>
[jira] [Commented] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files
[
https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961666#comment-15961666
]
Manoj Govindassamy commented on HDFS-11402:
---
Above unit test failures are not related to the patch.
> HDFS Snapshots should capture point-in-time copies of OPEN files
>
>
> Key: HDFS-11402
> URL: https://issues.apache.org/jira/browse/HDFS-11402
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11402.01.patch, HDFS-11402.02.patch,
> HDFS-11402.03.patch
>
>
> *Problem:*
> 1. When there are files being written and when HDFS Snapshots are taken in
> parallel, Snapshots do capture all these files, but these being written files
> in Snapshots do not have the point-in-time file length captured. That is,
> these open files are not frozen in HDFS Snapshots. These open files
> grow/shrink in length, just like the original file, even after the snapshot
> time.
> 2. At the time of File close or any other meta data modification operation on
> these files, HDFS reconciles the file length and records the modification in
> the last taken Snapshot. All the previously taken Snapshots continue to have
> those open Files with no modification recorded. So, all those previous
> snapshots end up using the final modification record in the last snapshot.
> Thus after the file close, file lengths in all those snapshots will end up
> same.
> Assume File1 is opened for write and a total of 1MB written to it. While the
> writes are happening, snapshots are taken in parallel.
> {noformat}
> |---Time---T1---T2-T3T4-->
> |---Snap1--Snap2-Snap3--->
> |---File1.open---write-write---close->
> {noformat}
> Then at time,
> T2:
> Snap1.File1.length = 0
> T3:
> Snap1.File1.length = 0
> Snap2.File1.length = 0
>
> T4:
> Snap1.File1.length = 1MB
> Snap2.File1.length = 1MB
> Snap3.File1.length = 1MB
> *Proposal*
> 1. At the time of taking Snapshot, {{SnapshotManager#createSnapshot}} can
> optionally request {{DirectorySnapshottableFeature#addSnapshot}} to freeze
> open files.
> 2. {{DirectorySnapshottableFeature#addSnapshot}} can consult with
> {{LeaseManager}} and get a list INodesInPath for all open files under the
> snapshot dir.
> 3. {{DirectorySnapshottableFeature#addSnapshot}} after the Snapshot creation,
> Diff creation and updating modification time, can invoke
> {{INodeFile#recordModification}} for each of the open files. This way, the
> Snapshot just taken will have a {{FileDiff}} with {{fileSize}} captured for
> each of the open files.
> 4. Above model follows the current Snapshot and Diff protocols and doesn't
> introduce any any disk formats. So, I don't think we will be needing any new
> FSImage Loader/Saver changes for Snapshots.
> 5. One of the design goals of HDFS Snapshot was ability to take any number of
> snapshots in O(1) time. LeaseManager though has all the open files with
> leases in-memory map, an iteration is still needed to prune the needed open
> files and then run recordModification on each of them. So, it will not be a
> strict O(1) with the above proposal. But, its going be a marginal increase
> only as the new order will be of O(open_files_under_snap_dir). In order to
> avoid HDFS Snapshots change in behavior for open files and avoid change in
> time complexity, this improvement can be made under a new config
> {{"dfs.namenode.snapshot.freeze.openfiles"}} which by default can be
> {{false}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files
[
https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961664#comment-15961664
]
Hadoop QA commented on HDFS-11402:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
788 unchanged - 4 fixed = 795 total (was 792) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
7s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11402 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862553/HDFS-11402.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux a1d2e9186ff6 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2aa8967 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19020/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
[jira] [Updated] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-11384: --- Attachment: HDFS-11384.003.patch > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, > HDFS-11384.001.patch, HDFS-11384.002.patch, HDFS-11384.003.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-11384: --- Attachment: (was: HDFS-11384.003.patch) > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, > HDFS-11384.001.patch, HDFS-11384.002.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11634) Optimize BlockIterator when interating starts in the middle.
[
https://issues.apache.org/jira/browse/HDFS-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-11634:
---
Attachment: HDFS-11634.001.patch
Simple optimization.
> Optimize BlockIterator when interating starts in the middle.
>
>
> Key: HDFS-11634
> URL: https://issues.apache.org/jira/browse/HDFS-11634
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Attachments: HDFS-11634.001.patch
>
>
> {{BlockManager.getBlocksWithLocations()}} needs to iterate blocks from a
> randomly selected {{startBlock}} index. It creates an iterator which points
> to the first block and then skips all blocks until {{startBlock}}. It is
> inefficient when DN has multiple storages. Instead of skipping blocks one by
> one we can skip entire storages. Should be more efficient on average.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11634) Optimize BlockIterator when interating starts in the middle.
[
https://issues.apache.org/jira/browse/HDFS-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-11634:
---
Status: Patch Available (was: Open)
> Optimize BlockIterator when interating starts in the middle.
>
>
> Key: HDFS-11634
> URL: https://issues.apache.org/jira/browse/HDFS-11634
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.5
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Attachments: HDFS-11634.001.patch
>
>
> {{BlockManager.getBlocksWithLocations()}} needs to iterate blocks from a
> randomly selected {{startBlock}} index. It creates an iterator which points
> to the first block and then skips all blocks until {{startBlock}}. It is
> inefficient when DN has multiple storages. Instead of skipping blocks one by
> one we can skip entire storages. Should be more efficient on average.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11634) Optimize BlockIterator when interating starts in the middle.
Konstantin Shvachko created HDFS-11634:
--
Summary: Optimize BlockIterator when interating starts in the
middle.
Key: HDFS-11634
URL: https://issues.apache.org/jira/browse/HDFS-11634
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.6.5
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
{{BlockManager.getBlocksWithLocations()}} needs to iterate blocks from a
randomly selected {{startBlock}} index. It creates an iterator which points to
the first block and then skips all blocks until {{startBlock}}. It is
inefficient when DN has multiple storages. Instead of skipping blocks one by
one we can skip entire storages. Should be more efficient on average.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files
[
https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11402:
--
Attachment: HDFS-11402.03.patch
Thanks for the review [~yzhangal]. Attached v03 patch with below review
comments addresses. Have added a new unit test for Snapshot Diff Report to
verify snapshot diffs for the snapshots which have open files captured. Kindly
take a look at the new patch revision.
bq. 1. We can put the parameters leaseManager and freezeOpenFiles together at
the API, since they are used together for an optional feature.
Done.
bq. 2. share common code in the two INodesInPath$fromINode methods
Done.
bq. 3. change method name isAncestor to isDescendent in INodesInPath
Done.
bq. 4. INODE_FILTER_WORKER_COUNT is only used in a single method, it's better
not to define it as public, and maybe we can just move it to the single method.
This count is used by unit tests as well. Have changed the scope of the
variable to package private instead of public.
bq. change getINodeWithLeases(final INodeDirectory restrictFilesFromDir)
to getINodesWithLease(final INodeDirectory ancestorDir)
and javadoc the behavior when ancestorDir is null or not-null
Done.
bq. optionally, possibly just use the above COUNT as a cap, and have a way to
dynamically decide how big the thread pool is, especially when the number of
files open for write is small. This can be consider in the future when needed.
Will take this for future.
bq. add a private method (like getINodesInLease to wrap
Done.
bq. 5. In hdfs-default.xml, add a note to describe that the file length
captured in snapshot for a file is what's recorded in NameNode, it may be
shorter than what the client has written. In order to capture the length the
client has written, the client need to call hflush/hsync on the file.
Done.
6. Suggest to add a test about snapshot diff.
Added a new unit test for Snapshot Diff Report
> HDFS Snapshots should capture point-in-time copies of OPEN files
>
>
> Key: HDFS-11402
> URL: https://issues.apache.org/jira/browse/HDFS-11402
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11402.01.patch, HDFS-11402.02.patch,
> HDFS-11402.03.patch
>
>
> *Problem:*
> 1. When there are files being written and when HDFS Snapshots are taken in
> parallel, Snapshots do capture all these files, but these being written files
> in Snapshots do not have the point-in-time file length captured. That is,
> these open files are not frozen in HDFS Snapshots. These open files
> grow/shrink in length, just like the original file, even after the snapshot
> time.
> 2. At the time of File close or any other meta data modification operation on
> these files, HDFS reconciles the file length and records the modification in
> the last taken Snapshot. All the previously taken Snapshots continue to have
> those open Files with no modification recorded. So, all those previous
> snapshots end up using the final modification record in the last snapshot.
> Thus after the file close, file lengths in all those snapshots will end up
> same.
> Assume File1 is opened for write and a total of 1MB written to it. While the
> writes are happening, snapshots are taken in parallel.
> {noformat}
> |---Time---T1---T2-T3T4-->
> |---Snap1--Snap2-Snap3--->
> |---File1.open---write-write---close->
> {noformat}
> Then at time,
> T2:
> Snap1.File1.length = 0
> T3:
> Snap1.File1.length = 0
> Snap2.File1.length = 0
>
> T4:
> Snap1.File1.length = 1MB
> Snap2.File1.length = 1MB
> Snap3.File1.length = 1MB
> *Proposal*
> 1. At the time of taking Snapshot, {{SnapshotManager#createSnapshot}} can
> optionally request {{DirectorySnapshottableFeature#addSnapshot}} to freeze
> open files.
> 2. {{DirectorySnapshottableFeature#addSnapshot}} can consult with
> {{LeaseManager}} and get a list INodesInPath for all open files under the
> snapshot dir.
> 3. {{DirectorySnapshottableFeature#addSnapshot}} after the Snapshot creation,
> Diff creation and updating modification time, can invoke
> {{INodeFile#recordModification}} for each of the open files. This way, the
> Snapshot just taken will have a {{FileDiff}} with {{fileSize}} captured for
> each of the open files.
> 4. Above model follows the current Snapshot and Diff protocols and doesn't
> introduce any any disk formats. So, I don't think we will be needing any new
> FSImage Loader/Saver changes for Snapshots.
> 5. One of the design goals of HDFS Snapshot was ability to take any number of
> snapshots in O(1) time. LeaseManager
[jira] [Commented] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus
[
https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961634#comment-15961634
]
Hadoop QA commented on HDFS-11565:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
30s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
48s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.protocolPB.TestPBHelper |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11565 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862546/HDFS-11565.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 914f51276636 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e8bdad7 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| mvninstall |
[jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[
https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961625#comment-15961625
]
Hadoop QA commented on HDFS-11384:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} HDFS-11384 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11384 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862552/HDFS-11384.003.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19019/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add option for balancer to disperse getBlocks calls to avoid NameNode's
> rpc.CallQueueLength spike
> -
>
> Key: HDFS-11384
> URL: https://issues.apache.org/jira/browse/HDFS-11384
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.7.3
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: balancer.day.png, balancer.week.png,
> HDFS-11384.001.patch, HDFS-11384.002.patch, HDFS-11384.003.patch
>
>
> When running balancer on hadoop cluster which have more than 3000 Datanodes
> will cause NameNode's rpc.CallQueueLength spike. We observed this situation
> could cause Hbase cluster failure due to RegionServer's WAL timeout.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-11384: --- Attachment: HDFS-11384.003.patch > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, > HDFS-11384.001.patch, HDFS-11384.002.patch, HDFS-11384.003.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-11384: --- Attachment: (was: HDFS-11384.003.patch) > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, > HDFS-11384.001.patch, HDFS-11384.002.patch, HDFS-11384.003.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-11384: --- Attachment: HDFS-11384.003.patch Here is a relatively simple patch, which restricts the number of RPC calls from Balancer to NN to 20 calls per second. 20 calls per second is a constant for now. It is chosen so that Balancer calls could not saturate NN's RPC queue based on metrics from a large cluster I was observing. LMK if people prefer it to be configurable. On a large cluster with 200 (default) dispatcher threads, and e.g. 500 underutilized DNs (sources) the initial 200 RPCs will be dispersed over 200 / 20 = 10 seconds. The remaining 300 RPCs should disperse organically as they subsequently reuse the same 200 threads from the pool. The patch has a unit test, which triggers the dispersion logic. > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, > HDFS-11384.001.patch, HDFS-11384.002.patch, HDFS-11384.003.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus
[
https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11565:
---
Attachment: HDFS-11565.002.patch
Patch attached. I also reverted the change to the PB tag numbers, which makes
it a little less incompatible.
> Use compact identifiers for built-in ECPolicies in HdfsFileStatus
> -
>
> Key: HDFS-11565
> URL: https://issues.apache.org/jira/browse/HDFS-11565
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11565.001.patch, HDFS-11565.002.patch
>
>
> Discussed briefly on HDFS-7337 with Kai Zheng. Quoting our convo:
> {quote}
> From looking at the protos, one other question I had is about the overhead of
> these protos when using the hardcoded policies. There are a bunch of strings
> and ints, which can be kind of heavy since they're added to each
> HdfsFileStatus. Should we make the built-in ones identified by purely an ID,
> with these fully specified protos used for the pluggable policies?
> {quote}
> {quote}
> Sounds like this could be considered separately because, either built-in
> policies or plugged-in polices, the full meta info is maintained either by
> the codes or in the fsimage persisted, so identifying them by purely an ID
> should works fine. If agree, we could refactor the codes you mentioned above
> separately.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch Available) Committed to trunk, thanks Kai and Wei-chiu for reviewing! > Move system erasure coding policies into hadoop-hdfs-client > --- > > Key: HDFS-11623 > URL: https://issues.apache.org/jira/browse/HDFS-11623 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch, > HDFS-11623.003.patch, HDFS-11623.004.patch, HDFS-11623.005.patch, > HDFS-11623.006.patch > > > This is a precursor to HDFS-11565. We need to move the set of system defined > EC policies out of the NameNode's ECPolicyManager into the hdfs-client module > so it can be referenced by the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961568#comment-15961568 ] Andrew Wang commented on HDFS-11623: Filed HADOOP-14293 for the TestFsDatasetImpl failure, others are unrelated and passed locally. Going to commit this shortly based on Wei-chiu's earlier +1. > Move system erasure coding policies into hadoop-hdfs-client > --- > > Key: HDFS-11623 > URL: https://issues.apache.org/jira/browse/HDFS-11623 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch, > HDFS-11623.003.patch, HDFS-11623.004.patch, HDFS-11623.005.patch, > HDFS-11623.006.patch > > > This is a precursor to HDFS-11565. We need to move the set of system defined > EC policies out of the NameNode's ECPolicyManager into the hdfs-client module > so it can be referenced by the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961554#comment-15961554
]
Hadoop QA commented on HDFS-11623:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
405 unchanged - 1 fixed = 406 total (was 406) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
15s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862535/HDFS-11623.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f8b5febc0026 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d298f73 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-11633) FSImage failover disables all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961548#comment-15961548 ] Hudson commented on HDFS-11633: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11551 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11551/]) HDFS-11633. FSImage failover disables all erasure coding policies. (wang: rev 203edc026c39224d1a5f762d05594c3757c6d908) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java > FSImage failover disables all erasure coding policies > -- > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11633.001.patch, HDFS-11633.002.patch, > HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage failover disables all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11633: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch Available) Nice find and fix [~jojochuang], committed to trunk! > FSImage failover disables all erasure coding policies > -- > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11633.001.patch, HDFS-11633.002.patch, > HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11633) FSImage failover disables all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961533#comment-15961533 ] Andrew Wang commented on HDFS-11633: +1 LGTM checking this in > FSImage failover disables all erasure coding policies > -- > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.002.patch, > HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11633) FSImage failover disables all erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961531#comment-15961531
]
Hadoop QA commented on HDFS-11633:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 68 unchanged - 1 fixed = 68 total (was 69) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11633 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862529/HDFS-11633.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c522a880580d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d298f73 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19016/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19016/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19016/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FSImage failover disables all erasure coding policies
> --
>
> Key: HDFS-11633
> URL: https://issues.apache.org/jira/browse/HDFS-11633
> Project: Hadoop HDFS
> Issue Type: Bug
> Components:
[jira] [Commented] (HDFS-10459) getTurnOffTip computes needed block incorrectly for threshold < 1 in b2.7
[
https://issues.apache.org/jira/browse/HDFS-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961501#comment-15961501
]
Hadoop QA commented on HDFS-10459:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 37 unchanged - 1 fixed = 41 total (was 38) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode |
| | hadoop.tracing.TestTracing |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10459 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862525/HDFS-10459.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4d27677c892a 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 56e81f2 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19015/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Attachment: HDFS-11623.006.patch Patch attached to fix TestOIV failure, have to use XOR for this. > Move system erasure coding policies into hadoop-hdfs-client > --- > > Key: HDFS-11623 > URL: https://issues.apache.org/jira/browse/HDFS-11623 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch, > HDFS-11623.003.patch, HDFS-11623.004.patch, HDFS-11623.005.patch, > HDFS-11623.006.patch > > > This is a precursor to HDFS-11565. We need to move the set of system defined > EC policies out of the NameNode's ECPolicyManager into the hdfs-client module > so it can be referenced by the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10687) Federation Membership State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961434#comment-15961434
]
Hadoop QA commented on HDFS-10687:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
4s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
1s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 30 new + 0 unchanged - 0 fixed = 30 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
56s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 9 new + 0
unchanged - 0 fixed = 9 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}118m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Useless control flow in
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization()
At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line
5067] |
| | Useless control flow in
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetExpiredRegistrationsRequestProto$Builder.maybeForceBuilderInitialization()
At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line
7082] |
| | Useless control flow in
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetNamespaceInfoRequestProto$Builder.maybeForceBuilderInitialization()
At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line
7420] |
| | Useless control flow in
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeHeartbeatResponseProto$Builder.maybeForceBuilderInitialization()
At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line
10335] |
| | Useless control flow in
[jira] [Commented] (HDFS-10999) Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats
[ https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961428#comment-15961428 ] Wei-Chiu Chuang commented on HDFS-10999: Thanks [~manojg] this is a pretty big patch and I am taking the time to review. bq. PS: DfsAdmin -report and WebUI are not updated to make use of the newer infrastructure. Probably after we finalize on this infra, I can take up the consumers separately in a new jira. HDFS-8196 is filed for displaying EC information. We can either extend that, or file a new jira explicitly for the metrics added in this one. We will need a new jira for DFSAdmin for sure. Thanks. > Introduce separate stats for Replicated and Erasure Coded Blocks apart from > the current Aggregated stats > > > Key: HDFS-10999 > URL: https://issues.apache.org/jira/browse/HDFS-10999 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-nice-to-have, supportability > Attachments: HDFS-10999.01.patch > > > Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic > term "low redundancy" to the old-fashioned "under replicated". But this term > is still being used in messages in several places, such as web ui, dfsadmin > and fsck. We should probably change them to avoid confusion. > File this jira to discuss it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage failover disables all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Attachment: HDFS-11633.002.patch Thanks [~andrew.wang]! Post 002 patch to make ECPM.clear() a non-op, until HDFS-7337 is in. Good to know that we have a follow up plan, but I am a little skeptical HDFS-7337 will be ready by alpha3. > FSImage failover disables all erasure coding policies > -- > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.002.patch, > HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961409#comment-15961409
]
Hadoop QA commented on HDFS-11623:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
405 unchanged - 1 fixed = 407 total (was 406) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
14s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862515/HDFS-11623.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c0b3ead79699 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 96cbb4f |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-10631) Federation State Store ZooKeeper implementation
[
https://issues.apache.org/jira/browse/HDFS-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961399#comment-15961399
]
Hadoop QA commented on HDFS-10631:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
8s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.getData(String,
Stat):in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.getData(String,
Stat): new String(byte[]) At StateStoreZooKeeperImpl.java:[line 466] |
| | Redundant nullcheck of znode, which is known to be non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.get(Class,
String) Redundant null check at StateStoreZooKeeperImpl.java:is known to be
non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.get(Class,
String) Redundant null check at StateStoreZooKeeperImpl.java:[line 231] |
| | Redundant nullcheck of znode, which is known to be non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.removeAll(Class)
Redundant null check at StateStoreZooKeeperImpl.java:is known to be non-null
in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.removeAll(Class)
Redundant null check at StateStoreZooKeeperImpl.java:[line 344] |
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| |
[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961386#comment-15961386 ] Wei-Chiu Chuang commented on HDFS-11623: Cool. That makes sense to me. Thanks! > Move system erasure coding policies into hadoop-hdfs-client > --- > > Key: HDFS-11623 > URL: https://issues.apache.org/jira/browse/HDFS-11623 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch, > HDFS-11623.003.patch, HDFS-11623.004.patch, HDFS-11623.005.patch > > > This is a precursor to HDFS-11565. We need to move the set of system defined > EC policies out of the NameNode's ECPolicyManager into the hdfs-client module > so it can be referenced by the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10459) getTurnOffTip computes needed block incorrectly for threshold < 1 in b2.7
[ https://issues.apache.org/jira/browse/HDFS-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HDFS-10459: --- Attachment: HDFS-10459.003.patch Either missed a test when I initially uploaded the trunk patch or it was added/modified since I put it up. Anyway, here's an updated patch for trunk. This patch applies to trunk and branch-2. > getTurnOffTip computes needed block incorrectly for threshold < 1 in b2.7 > - > > Key: HDFS-10459 > URL: https://issues.apache.org/jira/browse/HDFS-10459 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HDFS-10459.001.patch, HDFS-10459.002.patch, > HDFS-10459.003.patch, HDFS-10459-b2.7.002.patch, HDFS-10459-b2.7.003.patch > > > GetTurnOffTip overstates the number of blocks necessary to come out of safe > mode by 1 due to an arbitrary '+1' in the code. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10630) Federation State Store FS Implementation
[
https://issues.apache.org/jira/browse/HDFS-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961332#comment-15961332
]
Hadoop QA commented on HDFS-10630:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 402 unchanged - 0 fixed = 404 total (was 402) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
8s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Write to static field
org.apache.hadoop.hdfs.server.federation.store.StateStoreService.sharedService
from instance method
org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop()
At StateStoreService.java:from instance method
org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop()
At StateStoreService.java:[line 143] |
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10630 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862505/HDFS-10630-HDFS-10467-005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d7b1b7a0f04e 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10467 / 0e4661f |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Updated] (HDFS-10687) Federation Membership State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-10687: --- Attachment: HDFS-10687-HDFS-10467-002.patch Rebase HDFS-10467. > Federation Membership State Store internal API > -- > > Key: HDFS-10687 > URL: https://issues.apache.org/jira/browse/HDFS-10687 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10467-HDFS-10687-001.patch, > HDFS-10687-HDFS-10467-002.patch > > > The Federation Membership State encapsulates the information about the > Namenodes of each sub-cluster that are participating in Federation. The > information includes addresses for RPC, Web. This information is stored in > the State Store and later used by the Router to find data in the federation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10687) Federation Membership State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-10687: --- Attachment: (was: HDFS-10467-HDFS-10687-002.patch) > Federation Membership State Store internal API > -- > > Key: HDFS-10687 > URL: https://issues.apache.org/jira/browse/HDFS-10687 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10467-HDFS-10687-001.patch > > > The Federation Membership State encapsulates the information about the > Namenodes of each sub-cluster that are participating in Federation. The > information includes addresses for RPC, Web. This information is stored in > the State Store and later used by the Router to find data in the federation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961289#comment-15961289 ] Andrew Wang commented on HDFS-11623: Thanks for reviewing Wei-chiu! New patch to simplify the array as noted, and also to hopefully get a working precommit. Not sure why the build failed like that, it builds for me locally. Enabled policies work as you describe. The idea is that admins can restrict the set of policies used by the cluster. It doesn't affect already written files since they should still be readable. It doesn't affect already set policies to reduce user surprise, though we chould reconsider this and instead treat it similarly to min-replication. Thoughts? Agree that ECPManager looks quite simple now, but it'll be fleshed by the pluggable policy work at HDFS-7337. Kai sketched out the new responsibilities above. > Move system erasure coding policies into hadoop-hdfs-client > --- > > Key: HDFS-11623 > URL: https://issues.apache.org/jira/browse/HDFS-11623 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch, > HDFS-11623.003.patch, HDFS-11623.004.patch, HDFS-11623.005.patch > > > This is a precursor to HDFS-11565. We need to move the set of system defined > EC policies out of the NameNode's ECPolicyManager into the hdfs-client module > so it can be referenced by the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10631) Federation State Store ZooKeeper implementation
[ https://issues.apache.org/jira/browse/HDFS-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-10631: --- Attachment: HDFS-10631-HDFS-10467-002.patch Rebase HDFS-10467. > Federation State Store ZooKeeper implementation > --- > > Key: HDFS-10631 > URL: https://issues.apache.org/jira/browse/HDFS-10631 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10631-HDFS-10467-001.patch, > HDFS-10631-HDFS-10467-002.patch > > > State Store implementation using ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Attachment: HDFS-11623.005.patch > Move system erasure coding policies into hadoop-hdfs-client > --- > > Key: HDFS-11623 > URL: https://issues.apache.org/jira/browse/HDFS-11623 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch, > HDFS-11623.003.patch, HDFS-11623.004.patch, HDFS-11623.005.patch > > > This is a precursor to HDFS-11565. We need to move the set of system defined > EC policies out of the NameNode's ECPolicyManager into the hdfs-client module > so it can be referenced by the client. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10687) Federation Membership State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961275#comment-15961275
]
Hadoop QA commented on HDFS-10687:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-10687 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10687 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862513/HDFS-10467-HDFS-10687-002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19011/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Federation Membership State Store internal API
> --
>
> Key: HDFS-10687
> URL: https://issues.apache.org/jira/browse/HDFS-10687
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Inigo Goiri
>Assignee: Jason Kace
> Attachments: HDFS-10467-HDFS-10687-001.patch,
> HDFS-10467-HDFS-10687-002.patch
>
>
> The Federation Membership State encapsulates the information about the
> Namenodes of each sub-cluster that are participating in Federation. The
> information includes addresses for RPC, Web. This information is stored in
> the State Store and later used by the Router to find data in the federation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10687) Federation Membership State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-10687: --- Attachment: HDFS-10467-HDFS-10687-002.patch Rebased to HDFS-10467. Unit tests pending HDFS-10630. > Federation Membership State Store internal API > -- > > Key: HDFS-10687 > URL: https://issues.apache.org/jira/browse/HDFS-10687 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10467-HDFS-10687-001.patch, > HDFS-10467-HDFS-10687-002.patch > > > The Federation Membership State encapsulates the information about the > Namenodes of each sub-cluster that are participating in Federation. The > information includes addresses for RPC, Web. This information is stored in > the State Store and later used by the Router to find data in the federation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11633) FSImage failover disables all erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961250#comment-15961250
]
Andrew Wang commented on HDFS-11633:
Good catch! Yea, the clear is only supposed to clear things loaded from NN
metadata.
As an alternative fix, could we instead make the {{clear}} method a no-op, and
add a comment that we should only clear policies loaded from NN metadata?
HDFS-7337 will add pluggable policies that will be persisted and need to be
cleared.
> FSImage failover disables all erasure coding policies
> --
>
> Key: HDFS-11633
> URL: https://issues.apache.org/jira/browse/HDFS-11633
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, namenode
>Affects Versions: 3.0.0-alpha3
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch
>
>
> If NameNode fails to load the fsimage in the first NameNode metadata
> directory, it accidentally clears all enabled erasure coding policies in
> ErasureCodingPolicyManager.
> Even if the NameNode configures multiple fsimage metadata directory and it
> successfully loads the second one, enabled erasure coding policies are not
> restored.
> In the current implementation, we do not have a ErasureCodingPolicyManager
> section in fsimage, so a fsimage reload does not reload ECPM.
> The easiest fix before ECPM section is implemented, is don't clear ECPM when
> FSNamesystem is cleared.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11622) TraceId hardcoded to 0 in DataStreamer, correlation between multiple spans is lost
[
https://issues.apache.org/jira/browse/HDFS-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961241#comment-15961241
]
Andrew Purtell commented on HDFS-11622:
---
bq. I appreciate it if there are specific problems in downstream projects like
Phoenix, otherwise we should be conservative to fix core DFS code in
maintenance branch.
[~masatana] The spans in HDFS can have multiple parents because of batching in
the HBase WAL, which Phoenix uses as platform. An interesting case. I think if
this improvement is not committed to the shipping versions of Hadoop then
Phoenix tracing, based on HTrace, won't work correctly and cannot be reliably
used. HTrace is promising, but it needs to work correctly holistically with the
whole stack or is rendered moot.
> TraceId hardcoded to 0 in DataStreamer, correlation between multiple spans is
> lost
> --
>
> Key: HDFS-11622
> URL: https://issues.apache.org/jira/browse/HDFS-11622
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tracing
>Reporter: Karan Mehta
>
> In the {{run()}} method of {{DataStreamer}} class, the following code is
> written. {{parents\[0\]}} refer to the {{spanId}} of the parent span.
> {code}
> one = dataQueue.getFirst(); // regular data packet
> long parents[] = one.getTraceParents();
> if (parents.length > 0) {
> scope = Trace.startSpan("dataStreamer", new TraceInfo(0,
> parents[0]));
> // TODO: use setParents API once it's available from HTrace
> 3.2
> // scope = Trace.startSpan("dataStreamer", Sampler.ALWAYS);
> // scope.getSpan().setParents(parents);
> }
> {code}
> The {{scope}} starts a new TraceSpan with a traceId hardcoded to 0. Ideally
> it should be taken when {{currentPacket.addTraceParent(Trace.currentSpan())}}
> is invoked. This JIRA is to propose an additional long field inside the
> {{DFSPacket}} class which holds the parent {{traceId}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11622) TraceId hardcoded to 0 in DataStreamer, correlation between multiple spans is lost
[
https://issues.apache.org/jira/browse/HDFS-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961241#comment-15961241
]
Andrew Purtell edited comment on HDFS-11622 at 4/7/17 6:36 PM:
---
bq. I appreciate it if there are specific problems in downstream projects like
Phoenix, otherwise we should be conservative to fix core DFS code in
maintenance branch.
[~iwasakims] The spans in HDFS can have multiple parents because of batching in
the HBase WAL, which Phoenix uses as platform. An interesting case. I think if
this improvement is not committed to the shipping versions of Hadoop then
Phoenix tracing, based on HTrace, won't work correctly and cannot be reliably
used. HTrace is promising, but it needs to work correctly holistically with the
whole stack or is rendered moot.
was (Author: apurtell):
bq. I appreciate it if there are specific problems in downstream projects like
Phoenix, otherwise we should be conservative to fix core DFS code in
maintenance branch.
[~masatana] The spans in HDFS can have multiple parents because of batching in
the HBase WAL, which Phoenix uses as platform. An interesting case. I think if
this improvement is not committed to the shipping versions of Hadoop then
Phoenix tracing, based on HTrace, won't work correctly and cannot be reliably
used. HTrace is promising, but it needs to work correctly holistically with the
whole stack or is rendered moot.
> TraceId hardcoded to 0 in DataStreamer, correlation between multiple spans is
> lost
> --
>
> Key: HDFS-11622
> URL: https://issues.apache.org/jira/browse/HDFS-11622
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tracing
>Reporter: Karan Mehta
>
> In the {{run()}} method of {{DataStreamer}} class, the following code is
> written. {{parents\[0\]}} refer to the {{spanId}} of the parent span.
> {code}
> one = dataQueue.getFirst(); // regular data packet
> long parents[] = one.getTraceParents();
> if (parents.length > 0) {
> scope = Trace.startSpan("dataStreamer", new TraceInfo(0,
> parents[0]));
> // TODO: use setParents API once it's available from HTrace
> 3.2
> // scope = Trace.startSpan("dataStreamer", Sampler.ALWAYS);
> // scope.getSpan().setParents(parents);
> }
> {code}
> The {{scope}} starts a new TraceSpan with a traceId hardcoded to 0. Ideally
> it should be taken when {{currentPacket.addTraceParent(Trace.currentSpan())}}
> is invoked. This JIRA is to propose an additional long field inside the
> {{DFSPacket}} class which holds the parent {{traceId}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10630) Federation State Store FS Implementation
[ https://issues.apache.org/jira/browse/HDFS-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-10630: --- Attachment: HDFS-10630-HDFS-10467-005.patch Fixing unit tests, check style and find bugs. > Federation State Store FS Implementation > > > Key: HDFS-10630 > URL: https://issues.apache.org/jira/browse/HDFS-10630 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10630.001.patch, HDFS-10630.002.patch, > HDFS-10630-HDFS-10467-003.patch, HDFS-10630-HDFS-10467-004.patch, > HDFS-10630-HDFS-10467-005.patch > > > Interface to store the federation shared state across Routers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11618) Block Storage: Add Support for Direct I/O
[ https://issues.apache.org/jira/browse/HDFS-11618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961125#comment-15961125 ] Chen Liang commented on HDFS-11618: --- Committed to feature branch, thanks [~msingh] for the contribution! > Block Storage: Add Support for Direct I/O > - > > Key: HDFS-11618 > URL: https://issues.apache.org/jira/browse/HDFS-11618 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11618-HDFS-7240.001.patch, > HDFS-11618-HDFS-7240.002.patch > > > Currently Block Storage write the data to a leveldb Cache and then flushes > the data to the containers. This behavior should be configurable and support > should be added to write the data directly to the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11633) FSImage failover disables all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961052#comment-15961052 ] Wei-Chiu Chuang commented on HDFS-11633: Ping [~andrew.wang] you implemented fsimage fail over in HDFS-3277, would you like to chime in? [~zhz] would you also take a look too? > FSImage failover disables all erasure coding policies > -- > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage failover disables all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Summary: FSImage failover disables all erasure coding policies (was: FSImage loading fallback may disable all erasure coding policies ) > FSImage failover disables all erasure coding policies > -- > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961004#comment-15961004 ] Wei-Chiu Chuang commented on HDFS-11633: Okay now I understand where's the confusion. I probably used the wrong word. "Failover" is the right word. > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11626) Deprecate oiv_legacy tool
[ https://issues.apache.org/jira/browse/HDFS-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960871#comment-15960871 ] Kihwal Lee commented on HDFS-11626: --- bq. oiv_legacy only works for fsimage at or before Hadoop 2.4. We still support saving fsimage in the old writable format. This is of course only for analysis purposes, not meant for namenode consumption. If we decide to deprecate the tool, we might as well deprecate this capability. This format was kept because the memory requirement for processing it was low and independent of the size of the namespace. Often 128MB of heap is all it needs. The processing of PB format is not feasible if it requires over 10s of GB for a large name space we have. There are later improvements, but it still is inefficient compared to the writable-based image processing. If the future of human race depends on ditching this, we would deprecate it in 3 and remove it in 4. > Deprecate oiv_legacy tool > - > > Key: HDFS-11626 > URL: https://issues.apache.org/jira/browse/HDFS-11626 > Project: Hadoop HDFS > Issue Type: Task > Components: tools >Reporter: Wei-Chiu Chuang > > oiv_legacy only works for fsimage at or before Hadoop 2.4. I think we can > deprecate oiv_legacy in Hadoop 3 in preparation for final removal in Hadoop 4. > Or would people favor removing it in Hadoop 3? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960838#comment-15960838
]
Wei-Chiu Chuang commented on HDFS-11633:
[~drankye]
bq. I saw you only added some test codes and am not clear how it would fix the
reported issue.
My bad. HDFS-11633.test.patch is the test code with no fix.
HDFS-11633.001.patch has a "fix" where it does not clear ECPM in
{{FSNamesystem.clear()}}.
bq. The enabled erasure coding policies are from a configuration and then
maintained in the ErasureCodingPolicyManager. You mean there is some case that
the policies can be cleared?
Correct. FSImage loading fallback is just one of the triggers. It looks like
other events could also lead to the same issue.
> FSImage loading fallback may disable all erasure coding policies
> -
>
> Key: HDFS-11633
> URL: https://issues.apache.org/jira/browse/HDFS-11633
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, namenode
>Affects Versions: 3.0.0-alpha3
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch
>
>
> If NameNode fails to load the fsimage in the first NameNode metadata
> directory, it accidentally clears all enabled erasure coding policies in
> ErasureCodingPolicyManager.
> Even if the NameNode configures multiple fsimage metadata directory and it
> successfully loads the second one, enabled erasure coding policies are not
> restored.
> In the current implementation, we do not have a ErasureCodingPolicyManager
> section in fsimage, so a fsimage reload does not reload ECPM.
> The easiest fix before ECPM section is implemented, is don't clear ECPM when
> FSNamesystem is cleared.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960830#comment-15960830
]
Wei Zhou commented on HDFS-7343:
Thanks [~rakeshr] for reviewing the design and thoughtful comments.
{quote}
I hope these tables will be deleted after performing the aggregation functions
{quote}
Sorry for not making it clear that tables will be deleted after performing
aggregation. Through aggregation, the total number of tables in SSM are
controllable.
{quote}
minimize the number of time spec tables, how about capturing...
{quote}
Yes, it can reduce the number of tables and the costs of across table query
operations. It can be good for situations, but there can be some issues either:
- It increases the memory to store these data. For every polling from NN and
each file, you have to store an {{access_time}}, so the table looks like in the
following one (assume 5s for next polling). In fact, the {{access_time}} for
files returned from the same polling are the same, so about 1/3 memory
consumption can be saved. Also, only {{access_time}} may be not enough for
determine whether it is in the given time interval, an {{end_time}} is also
needed, so totally it will bring 100% memory overhead.
||acess_time||fid||count||
|sec-2017-03-31-12-59-05|1|3|
|sec-...|...|...|
|sec-2017-03-31-12-59-35|1|2|
|sec-2017-03-31-12-59-35|3|7|
|sec-2017-03-31-12-59-40|3|1|
|sec-2017-03-31-12-59-45|3|1|
|sec-2017-03-31-12-59-45|2|1|
|sec-2017-03-31-12-59-50|3|3|
- It also can bring performance issue. For example, if a rule only cares the
last 5s data, then the whole large table will be scanned in order to filter out
the records needed.
{quote}
maintain separate tables per units of time
{quote}
Yes, it's much better to do this if data are tracked in the way mentioned above.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: access_count_tables.jpg,
> HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg, tables_in_ssm.xlsx
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size
[
https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960624#comment-15960624
]
Hadoop QA commented on HDFS-9342:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-9342 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862471/HDFS-9342.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e1a96e4d790e 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ad24464 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19009/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19009/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure coding: client should update and commit block based on acknowledged
> size
>
>
> Key: HDFS-9342
> URL: https://issues.apache.org/jira/browse/HDFS-9342
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Zhe Zhang
>Assignee: SammiChen
>Priority:
[jira] [Updated] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size
[
https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SammiChen updated HDFS-9342:
Status: Patch Available (was: Open)
> Erasure coding: client should update and commit block based on acknowledged
> size
>
>
> Key: HDFS-9342
> URL: https://issues.apache.org/jira/browse/HDFS-9342
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Zhe Zhang
>Assignee: SammiChen
>Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9342.01.patch, HDFS-9342.02.patch,
> HDFS-9342.03.patch, HDFS-9342.04.patch
>
>
> For non-EC files, we have:
> {code}
> protected ExtendedBlock block; // its length is number of bytes acked
> {code}
> For EC files, the size of {{DFSStripedOutputStream#currentBlockGroup}} is
> incremented in {{writeChunk}} without waiting for ack. And both
> {{updatePipeline}} and {{commitBlock}} are based on size of
> {{currentBlockGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size
[
https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SammiChen updated HDFS-9342:
Attachment: HDFS-9342.04.patch
Refine the v3 patch
> Erasure coding: client should update and commit block based on acknowledged
> size
>
>
> Key: HDFS-9342
> URL: https://issues.apache.org/jira/browse/HDFS-9342
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Zhe Zhang
>Assignee: SammiChen
>Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9342.01.patch, HDFS-9342.02.patch,
> HDFS-9342.03.patch, HDFS-9342.04.patch
>
>
> For non-EC files, we have:
> {code}
> protected ExtendedBlock block; // its length is number of bytes acked
> {code}
> For EC files, the size of {{DFSStripedOutputStream#currentBlockGroup}} is
> incremented in {{writeChunk}} without waiting for ack. And both
> {{updatePipeline}} and {{commitBlock}} are based on size of
> {{currentBlockGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960538#comment-15960538
]
Hadoop QA commented on HDFS-11633:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 68 unchanged - 1 fixed = 68 total (was 69) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.server.namenode.TestStartup |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11633 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862446/HDFS-11633.test.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ea36ffec0d1e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ad24464 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19008/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19008/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FSImage loading fallback may disable all erasure coding policies
> -
>
>
[jira] [Commented] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960465#comment-15960465 ] Kai Zheng commented on HDFS-11633: -- Hi [~jojochuang], I saw you only added some test codes and am not clear how it would fix the reported issue. The enabled erasure coding policies are from a configuration and then maintained in the ErasureCodingPolicyManager. You mean there is some case that the policies can be cleared? > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Status: Patch Available (was: Open) > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Labels: hdfs-ec-3.0-must-do (was: ) > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11622) TraceId hardcoded to 0 in DataStreamer, correlation between multiple spans is lost
[
https://issues.apache.org/jira/browse/HDFS-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960420#comment-15960420
]
Masatake Iwasaki commented on HDFS-11622:
-
bq. If you want, I can submit a patch for this one.
I appreciate it if there are specific problems in downstream projects like
Phoenix, otherwise we should be conservative to fix core DFS code in
maintenance branch.
> TraceId hardcoded to 0 in DataStreamer, correlation between multiple spans is
> lost
> --
>
> Key: HDFS-11622
> URL: https://issues.apache.org/jira/browse/HDFS-11622
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tracing
>Reporter: Karan Mehta
>
> In the {{run()}} method of {{DataStreamer}} class, the following code is
> written. {{parents\[0\]}} refer to the {{spanId}} of the parent span.
> {code}
> one = dataQueue.getFirst(); // regular data packet
> long parents[] = one.getTraceParents();
> if (parents.length > 0) {
> scope = Trace.startSpan("dataStreamer", new TraceInfo(0,
> parents[0]));
> // TODO: use setParents API once it's available from HTrace
> 3.2
> // scope = Trace.startSpan("dataStreamer", Sampler.ALWAYS);
> // scope.getSpan().setParents(parents);
> }
> {code}
> The {{scope}} starts a new TraceSpan with a traceId hardcoded to 0. Ideally
> it should be taken when {{currentPacket.addTraceParent(Trace.currentSpan())}}
> is invoked. This JIRA is to propose an additional long field inside the
> {{DFSPacket}} class which holds the parent {{traceId}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960373#comment-15960373
]
Wei-Chiu Chuang edited comment on HDFS-11623 at 4/7/17 7:38 AM:
Overall looks good to me, so I am +1. I just want to point out a few things for
discussion.
# After this change, ErasureCodingPolicyManager becomes a little redundancy. It
is only useful for {{DistributedFileSystem#setErasureCodingPolicy()}} and
{{DistributedFileSystem#getErasureCodingPolicies()}}.
# Also, the concept of enabled policies is a little foreign to me. If the
system has a file with ec policy A, but later the system disables the ec policy
A, that file is still fine. It's just that no more new files can be created
with ec policy A.
# My editor told me the following code:
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
new ErasureCodingPolicy[]{
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4,
SYS_POLICY5}));
{code}
can be simplified to
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4, SYS_POLICY5));
{code}
By the way, I found a critical issue that I will post later. HDFS-11633. This
is unrelated to this patch but I found it while reviewing it.
was (Author: jojochuang):
Overall looks good to me, so I am +1. I just want to point out a few things for
discussion.
# After this change, ErasureCodingPolicyManager becomes a little redundancy. It
is only useful for {{DistributedFileSystem#setErasureCodingPolicy()}} and
{{DistributedFileSystem#getErasureCodingPolicies()}}.
# Also, the concept of enabled policies is a little foreign to me. If the
system has a file with ec policy A, but later the system disables the ec policy
A, that file is still fine. It's just that no more new files can be created
with ec policy A.
# My editor told me the following code:
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
new ErasureCodingPolicy[]{
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4,
SYS_POLICY5}));
{code}
can be simplified to
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4, SYS_POLICY5));
{code}
By the way, I found a critical issue that I will post later.
> Move system erasure coding policies into hadoop-hdfs-client
> ---
>
> Key: HDFS-11623
> URL: https://issues.apache.org/jira/browse/HDFS-11623
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch,
> HDFS-11623.003.patch, HDFS-11623.004.patch
>
>
> This is a precursor to HDFS-11565. We need to move the set of system defined
> EC policies out of the NameNode's ECPolicyManager into the hdfs-client module
> so it can be referenced by the client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Attachment: HDFS-11633.test.patch > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Attachment: (was: HDFS-11633.001.patch) > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Attachment: HDFS-11633.test.patch > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Attachments: HDFS-11633.001.patch, HDFS-11633.test.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960373#comment-15960373
]
Wei-Chiu Chuang edited comment on HDFS-11623 at 4/7/17 7:30 AM:
Overall looks good to me, so I am +1. I just want to point out a few things for
discussion.
# After this change, ErasureCodingPolicyManager becomes a little redundancy. It
is only useful for {{DistributedFileSystem#setErasureCodingPolicy()}} and
{{DistributedFileSystem#getErasureCodingPolicies()}}.
# Also, the concept of enabled policies is a little foreign to me. If the
system has a file with ec policy A, but later the system disables the ec policy
A, that file is still fine. It's just that no more new files can be created
with ec policy A.
# My editor told me the following code:
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
new ErasureCodingPolicy[]{
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4,
SYS_POLICY5}));
{code}
can be simplified to
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4, SYS_POLICY5));
{code}
By the way, I found a critical issue that I will post later.
was (Author: jojochuang):
Overall looks good to me. I just want to point out a few things for discussion.
# After this change, ErasureCodingPolicyManager becomes a little redundancy. It
is only useful for {{DistributedFileSystem#setErasureCodingPolicy()}} and
{{DistributedFileSystem#getErasureCodingPolicies()}}.
# Also, the concept of enabled policies is a little foreign to me. If the
system has a file with ec policy A, but later the system disables the ec policy
A, that file is still fine. It's just that no more new files can be created
with ec policy A.
# My editor told me the following code:
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
new ErasureCodingPolicy[]{
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4,
SYS_POLICY5}));
{code}
can be simplified to
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4, SYS_POLICY5));
{code}
By the way, I found a critical issue that I will post later.
> Move system erasure coding policies into hadoop-hdfs-client
> ---
>
> Key: HDFS-11623
> URL: https://issues.apache.org/jira/browse/HDFS-11623
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch,
> HDFS-11623.003.patch, HDFS-11623.004.patch
>
>
> This is a precursor to HDFS-11565. We need to move the set of system defined
> EC policies out of the NameNode's ECPolicyManager into the hdfs-client module
> so it can be referenced by the client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage loading fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Summary: FSImage loading fallback may disable all erasure coding policies (was: FSImage load fallback may disable all erasure coding policies ) > FSImage loading fallback may disable all erasure coding policies > - > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Attachments: HDFS-11633.001.patch, HDFS-11633.001.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11633) FSImage load fallback may disable all erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11633: --- Attachment: HDFS-11633.001.patch patch v001: a simple fix: do not clear enabled ec policies when FSNamesystem is cleared. FSNamesystem is cleared when a fsimage can not be loaded. > FSImage load fallback may disable all erasure coding policies > -- > > Key: HDFS-11633 > URL: https://issues.apache.org/jira/browse/HDFS-11633 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, namenode >Affects Versions: 3.0.0-alpha3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Critical > Attachments: HDFS-11633.001.patch, HDFS-11633.001.patch > > > If NameNode fails to load the fsimage in the first NameNode metadata > directory, it accidentally clears all enabled erasure coding policies in > ErasureCodingPolicyManager. > Even if the NameNode configures multiple fsimage metadata directory and it > successfully loads the second one, enabled erasure coding policies are not > restored. > In the current implementation, we do not have a ErasureCodingPolicyManager > section in fsimage, so a fsimage reload does not reload ECPM. > The easiest fix before ECPM section is implemented, is don't clear ECPM when > FSNamesystem is cleared. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960373#comment-15960373
]
Wei-Chiu Chuang edited comment on HDFS-11623 at 4/7/17 6:57 AM:
Overall looks good to me. I just want to point out a few things for discussion.
# After this change, ErasureCodingPolicyManager becomes a little redundancy. It
is only useful for {{DistributedFileSystem#setErasureCodingPolicy()}} and
{{DistributedFileSystem#getErasureCodingPolicies()}}.
# Also, the concept of enabled policies is a little foreign to me. If the
system has a file with ec policy A, but later the system disables the ec policy
A, that file is still fine. It's just that no more new files can be created
with ec policy A.
# My editor told me the following code:
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
new ErasureCodingPolicy[]{
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4,
SYS_POLICY5}));
{code}
can be simplified to
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4, SYS_POLICY5));
{code}
By the way, I found a critical issue that I will post later.
was (Author: jojochuang):
Overall looks good to me. I just want to point out a few things for discussion.
# After this change, ErasureCodingPolicyManager becomes a little redundancy. It
is only useful for {{DistributedFileSystem#setErasureCodingPolicy()}} and
{{DistributedFileSystem#getErasureCodingPolicies()}}. That is to say, if the
system has a file with ec policy A, but later the system disables the ec policy
A, that file is still fine. It's just that no more new files can be created
with ec policy A.
My editor told me the following code:
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
new ErasureCodingPolicy[]{
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4,
SYS_POLICY5}));
{code}
can be simplified to
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4, SYS_POLICY5));
{code}
By the way, I found a critical issue that I will post later.
> Move system erasure coding policies into hadoop-hdfs-client
> ---
>
> Key: HDFS-11623
> URL: https://issues.apache.org/jira/browse/HDFS-11623
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch,
> HDFS-11623.003.patch, HDFS-11623.004.patch
>
>
> This is a precursor to HDFS-11565. We need to move the set of system defined
> EC policies out of the NameNode's ECPolicyManager into the hdfs-client module
> so it can be referenced by the client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960373#comment-15960373
]
Wei-Chiu Chuang commented on HDFS-11623:
Overall looks good to me. I just want to point out a few things for discussion.
# After this change, ErasureCodingPolicyManager becomes a little redundancy. It
is only useful for {{DistributedFileSystem#setErasureCodingPolicy()}} and
{{DistributedFileSystem#getErasureCodingPolicies()}}. That is to say, if the
system has a file with ec policy A, but later the system disables the ec policy
A, that file is still fine. It's just that no more new files can be created
with ec policy A.
My editor told me the following code:
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
new ErasureCodingPolicy[]{
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4,
SYS_POLICY5}));
{code}
can be simplified to
{code}
private static final List SYS_POLICIES =
Collections.unmodifiableList(Arrays.asList(
SYS_POLICY1, SYS_POLICY2, SYS_POLICY3, SYS_POLICY4, SYS_POLICY5));
{code}
By the way, I found a critical issue that I will post later.
> Move system erasure coding policies into hadoop-hdfs-client
> ---
>
> Key: HDFS-11623
> URL: https://issues.apache.org/jira/browse/HDFS-11623
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HDFS-11623.001.patch, HDFS-11623.002.patch,
> HDFS-11623.003.patch, HDFS-11623.004.patch
>
>
> This is a precursor to HDFS-11565. We need to move the set of system defined
> EC policies out of the NameNode's ECPolicyManager into the hdfs-client module
> so it can be referenced by the client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
