[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997196#comment-15997196 ] Andrew Wang commented on HDFS-11643: Fair enough. We should try revising this once HDFS-11644 goes in, since it offers a clean way of querying the stream's capabilities without the additional RPC. I'll file a follow-on JIRA. Let's get this in, thanks Sammi! Will commit shortly. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch, > HDFS-11643.006.patch, HDFS-11643.007.patch, HDFS-11643.008.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996052#comment-15996052
]
SammiChen commented on HDFS-11643:
--
My consideration is Id lock file is not only used for {{Balancer}}, but also
used for {{Mover}}, might be used for other tool in future, so add the check
in production code is a once-for-all solution. On the other hand, Id lock file
path is an private data of {{Balancer}}, it's not exposed to outside. If put
the check in unit test, Id file path will be hard coded. Once the Id file path
changed, the unit test will fail. So I would prefer put the check in
production code unless there is severe impact.
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch,
> HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch,
> HDFS-11643.006.patch, HDFS-11643.007.patch, HDFS-11643.008.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995848#comment-15995848
]
Andrew Wang commented on HDFS-11643:
I see the check was added to NameNodeConnector, can it be added to TestBalancer
instead, for instance in {{runBalancer}}? I'd prefer that we not add the check
to the production code.
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch,
> HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch,
> HDFS-11643.006.patch, HDFS-11643.007.patch, HDFS-11643.008.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994572#comment-15994572
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
27s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
30s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 6s{color} | {color:orange} root: The patch generated 2 new + 500 unchanged
- 2 fixed = 502 total (was 502) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
55s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.namenode.TestStartup |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994194#comment-15994194 ] Andrew Wang commented on HDFS-11643: Woops, I missed that, thanks Sammi. Still, given that just the TestBalancer change works without the rest of the patch applied, we should do an explicit check that the balancer's lock file is replicated and not EC. New tests should exhibit "fail before, works after" behavior. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch, > HDFS-11643.006.patch, HDFS-11643.007.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994164#comment-15994164
]
SammiChen commented on HDFS-11643:
--
Hi [~andrew.wang], the changes to use the new shouldReplicate builder API is
in {{NameNodeConnector}} class. It enforce the Id lock file to be a replicated
file instead of striped file. So even without the HDFS-11718 commit, the new
unit test should work.
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch,
> HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch,
> HDFS-11643.006.patch, HDFS-11643.007.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993894#comment-15993894 ] Andrew Wang commented on HDFS-11643: Hi Sammi, I noticed that the 007 patch doesn't include any changes to the balancer to use the new shouldReplicate builder API. The new unit test changes work without the rest of the patch applied because we just committed HDFS-11718 so EC files don't throw an exception on hflush/hsync. Can we add an assert to check that the balancer's lock file is replicated? Thanks. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch, > HDFS-11643.006.patch, HDFS-11643.007.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992825#comment-15992825 ] SammiChen commented on HDFS-11643: -- Double checked the failed unit test. They also failed without the patch applied. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch, > HDFS-11643.006.patch, HDFS-11643.007.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992604#comment-15992604
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
43s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
8s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 16m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 10s{color} | {color:orange} root: The patch generated 2 new + 501 unchanged
- 2 fixed = 503 total (was 503) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
2s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.TestStartup |
| | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11643 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12865879/HDFS-11643.007.patch |
| Optional Tests | asflicense compile javac
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991964#comment-15991964 ] SammiChen commented on HDFS-11643: -- Thanks [~jojochuang] & [~andrew.wang] for review the patch! CreateFlag is a good idea. I will upload a new patch later. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch, > HDFS-11643.006.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991459#comment-15991459 ] Andrew Wang commented on HDFS-11643: Thanks for reviewing Wei-chiu, Despite what downstreams may do, DFSClient is a private API, so we're allowed to change it. However, good point about CreateFlag, we definitely should avoid adding new boolean parameters. I missed this during my review. Sammi, could you update the patch? I think this only affects the wire format, let's leave the builder API as-is. I left a comment on HADOOP-14365 about a proposed enhancement for the flags. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch, > HDFS-11643.006.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988979#comment-15988979 ] Wei-Chiu Chuang commented on HDFS-11643: HI [~Sammi] Thanks for working on this patch. I know Andrew has basically +1 the last patch, but I'd like to ask for a second thought: I think you need to mark this jira as an incompatible change, because the method signature of DFSClient.create() is changed. I also wonder if it's possible to use CreateFlag, instead of adding a new parameter shouldReplicate for the same purpose. This will also avoid the incompatibility issue, plus, it's really not my taste to have so many parameters. I think it's a good idea to think twice about changing the signature of a public API. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch, > HDFS-11643.006.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979928#comment-15979928
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
28s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-hdfs-project: The patch generated 10 new
+ 1225 unchanged - 10 fixed = 1235 total (was 1235) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HDFS-11643 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864625/HDFS-11643.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux cd8268b7dd77 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fda86ef |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979794#comment-15979794
]
SammiChen commented on HDFS-11643:
--
Thanks Andrew for the reminder! TestBalancer issue is fixed. It seems some
unit test doesn't work well when Write2IdFile is enabled.
{quote} Not sure if this is related, but running TestBalancer also makes this
file in my source tree, rather than in a target folder:
hadoop-hdfs-project/hadoop-hdfs/include-hosts-file
{quote}
I not quite understand this comment.
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch,
> HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch,
> HDFS-11643.006.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979355#comment-15979355 ] Andrew Wang commented on HDFS-11643: Not sure if this is related, but running TestBalancer also makes this file in my source tree, rather than in a target folder: hadoop-hdfs-project/hadoop-hdfs/include-hosts-file > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979337#comment-15979337 ] Andrew Wang commented on HDFS-11643: I think the TestBalancer issues are related, could you take a look? > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978189#comment-15978189
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
5s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 13s{color} | {color:orange} hadoop-hdfs-project: The patch generated 10 new
+ 1225 unchanged - 10 fixed = 1235 total (was 1235) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HDFS-11643 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864426/HDFS-11643.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 73b130d6613d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978027#comment-15978027 ] SammiChen commented on HDFS-11643: -- Upload a new Patch. Thanks, Andrew! > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch, HDFS-11643.005.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977205#comment-15977205
]
Andrew Wang commented on HDFS-11643:
Looks pretty close, wanted to clarify a few renames, +1 pending these:
* Recommend rename {{enforceReplicate}} to just {{replicate}}
* Many variables named {{asReplicatedFile}} still, please update these too,
maybe {{shouldReplicate}} instead like the getter name? Rename the proto field
too.
Really appreciate your work here Sammi!
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch,
> HDFS-11643.003.patch, HDFS-11643.004.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976270#comment-15976270 ] SammiChen commented on HDFS-11643: -- Double checked the issues reported by findbugs, asflicense and unit test. It seems all are irrelevant. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976174#comment-15976174
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
25s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
41s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-hdfs-project: The patch generated 10 new
+ 1225 unchanged - 10 fixed = 1235 total (was 1235) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
24s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HDFS-11643 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864195/HDFS-11643.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 7d9d0c7a119b 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 74a7238 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976049#comment-15976049 ] SammiChen commented on HDFS-11643: -- Uploaded a new patch, improved based on Andrew's suggestion. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch, > HDFS-11643.003.patch, HDFS-11643.004.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975451#comment-15975451
]
Andrew Wang commented on HDFS-11643:
Hi Sammi, took a deeper look at this, mostly just nitty and API stuff:
HdfsDataOutputStreamBuilder:
* We can simplify and rename the setter {{enforceAsReplicatedFile(Boolean)}} to
just be {{replicate()}} with no parameters, since the default is false. I think
this could also just be a primitive boolean, since the {{dfs.create}} takes a
primitive boolean rather than a nullable Boolean.
* asReplicatedFile, recommend we rename to {{shouldReplicate}} and make the
getter protected
* We also should add some Javadoc to {{replicateFile}} and {{setEcPolicyName}}
to explain what they do and that they are exclusive.
I'd prefer to be more consistent about {{ecPolicyName}} and {{asReplicated}}
being exclusive parameters and throwing an exception. We fixed the builder, but
a few other places I noticed:
* ClientProtocol: docs say that ecPolicyName is ignored, instead, let's make it
clear that it's invalid to set {{asReplicated}} to true and also a non-null
{{ecPolicyName}}.
* Builder#build: Please add some additional help text for the user when the
Precondition check fires
* FSNamesystem#startFileInt: Let's check exclusivity here as well, can do it
outside the write lock where we do the invalid path check.
Tests:
* Note that the DiskBalancer is different from the Balancer. We should be
adding a test to TestBalancer instead.
Recommend reformatting these chained calls in TestErasureCodingPolicies like
this for clarity:
{noformat}
fs.newFSDataOutputStreamBuilder(filePath).setEcPolicyName(ecPolicyName)
.enforceAsReplicatedFile(false).build().close();
{noformat}
{noformat}
fs.newFSDataOutputStreamBuilder(filePath)
.setEcPolicyName(ecPolicyName)
.enforceAsReplicatedFile(false)
.build()
.close();
{noformat}
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch,
> HDFS-11643.003.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974668#comment-15974668
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} hadoop-hdfs-project: The patch generated 10 new
+ 1028 unchanged - 10 fixed = 1038 total (was 1038) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 2s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HDFS-11643 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863990/HDFS-11643.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux fed952f08959 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6b015d0 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19142/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974422#comment-15974422
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-hdfs-project: The patch generated 11 new
+ 1029 unchanged - 10 fixed = 1040 total (was 1039) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestLease |
| |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HDFS-11643 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863974/HDFS-11643.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux ea4da6ac5c2b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8c81a16 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974315#comment-15974315 ] SammiChen commented on HDFS-11643: -- Thanks [~andrew.wang] for review and comments! A new patch is uploaded. All the concerns are addressed. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11643.001.patch, HDFS-11643.002.patch > > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970974#comment-15970974
]
Hadoop QA commented on HDFS-11643:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-11643 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11643 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863634/HDFS-11643.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19105/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11643.001.patch
>
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[
https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966691#comment-15966691
]
Andrew Wang commented on HDFS-11643:
Agree, at least for the balancer, I think for now we only need a new boolean
parameter like you say.
This also relates to HDFS-11644. If DFSStripedOutputStream no longer implements
Syncable, then the Balancer's FSDataOutputStream#hflush will fallback to just
doing a flush. For the balancer, I think we'd still prefer writing a replicated
file and doing a real hflush, since otherwise {{write2IdFile}} won't function
correctly.
> Balancer fencing fails when writing erasure coded lock file
> ---
>
> Key: HDFS-11643
> URL: https://issues.apache.org/jira/browse/HDFS-11643
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
>
> At startup, the balancer writes its hostname to the lock file and calls
> hflush(). hflush is not supported for EC files, so this fails when the entire
> filesystem is erasure coded.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965269#comment-15965269 ] SammiChen commented on HDFS-11643: -- Hi [~andrew.wang], yea, we need face this a "replicated" EC policy now. Let me first clear understand the requirements. So to your knowledge, can a file create API which forces use default replication solve all the issues we met so far and all foreseeable issues? Or should we also provide a way to set a "replicated" EC policy on directory? If only a create API is required, I would suggest not introduce the "replicated" EC policy concept, we can add a Boolean parameter to create function to enforce the file to be replication file. If we also need to provide set an directory back to replication, other than inherit its parent's EC policy, then "replicated" EC policy is a must. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964941#comment-15964941 ] Andrew Wang commented on HDFS-11643: Heads up [~Sammi], this relates to our discussion about a "replicated" EC policy. We could use it to force the balancer lock file to be a replicated file. > Balancer fencing fails when writing erasure coded lock file > --- > > Key: HDFS-11643 > URL: https://issues.apache.org/jira/browse/HDFS-11643 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > At startup, the balancer writes its hostname to the lock file and calls > hflush(). hflush is not supported for EC files, so this fails when the entire > filesystem is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
