[jira] [Commented] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006331#comment-17006331
]
Ayush Saxena commented on HDFS-14578:
-
I had the WIP patch with me since couple of months,
Uploaded to depict the idea, Guarded the logic by a configuration, as of now in
WIP it is default to true, so as to accommodate test in the same class.
[~weichiu] [~vinayakumarb] Can you give a check once.
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14578) AvailableSpaceBlockPlacementPolicy always prefers local node
[
https://issues.apache.org/jira/browse/HDFS-14578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14578:
Attachment: HDFS-14578-WIP-01.patch
> AvailableSpaceBlockPlacementPolicy always prefers local node
>
>
> Key: HDFS-14578
> URL: https://issues.apache.org/jira/browse/HDFS-14578
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14578-WIP-01.patch
>
>
> It looks like AvailableSpaceBlockPlacementPolicy prefers local disk just like
> in the BlockPlacementPolicyDefault
>
> As Yongjun mentioned in
> [HDFS-8131|https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=16558739&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16558739],
>
> {quote}Class AvailableSpaceBlockPlacementPolicy extends
> BlockPlacementPolicyDefault. But it doesn't change the behavior of choosing
> the first node in BlockPlacementPolicyDefault, so even with this new feature,
> the local DN is always chosen as the first DN (of course when it is not
> excluded), and the new feature only changes the selection of the rest of the
> two DNs.
> {quote}
> I'm file this Jira as I groom Cloudera's internal Jira and found this
> unreported issue. We do have a customer hitting this problem. I don't have a
> fix, but thought it would be beneficial to report it to Apache Jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15087) RBF: Balance/Rename across federation namespaces
[
https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006329#comment-17006329
]
Yiqun Lin commented on HDFS-15087:
--
Hi [~LiJinglun], it's an interesting feature! One major concern from me:
{quote}The HFR is splitted into 5 steps: Prepare, SaveTree, GraftTree, HardLink
and Finish...
{quote}
We have 5 steps here and I see only some steps like some metadata change in dst
namespace we hold the lock. So how can we ensure that source directory not
being changed during that time? Or we recommend use HRF only for small paths
that won't have frequent change? This should be the key point that users want
to know. People will see inconsistent files between src/dst namespaces once
this cannot be promised.
> RBF: Balance/Rename across federation namespaces
>
>
> Key: HDFS-15087
> URL: https://issues.apache.org/jira/browse/HDFS-15087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jinglun
>Priority: Major
> Attachments: HFR_Rename Across Federation Namespaces.pdf
>
>
> The Xiaomi storage team has developed a new feature called HFR(HDFS
> Federation Rename) that enables us to do balance/rename across federation
> namespaces. The idea is to first move the meta to the dst NameNode and then
> link all the replicas. It has been working in our largest production cluster
> for 2 months. We use it to balance the namespaces. It turns out HFR is fast
> and flexible. The detail could be found in the design doc.
> Looking forward to a lively discussion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15091) Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock
[ https://issues.apache.org/jira/browse/HDFS-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15091: Attachment: HDFS-15091-02.patch > Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock > > > Key: HDFS-15091 > URL: https://issues.apache.org/jira/browse/HDFS-15091 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15091-01.patch, HDFS-15091-02.patch > > > As of now all API check superuser before taking lock, Similarly can be done > for the cache commands and setQuota. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15091) Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock
[
https://issues.apache.org/jira/browse/HDFS-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006317#comment-17006317
]
Hadoop QA commented on HDFS-15091:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
21s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to pc in
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.modifyCachePool(FSNamesystem,
CacheManager, CachePoolInfo, boolean) At
FSNDNCacheOp.java:org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.modifyCachePool(FSNamesystem,
CacheManager, CachePoolInfo, boolean) At FSNDNCacheOp.java:[line 89] |
| | Dead store to pc in
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.removeCachePool(FSNamesystem,
CacheManager, String, boolean) At
FSNDNCacheOp.java:org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.removeCachePool(FSNamesystem,
CacheManager, String, boolean) At FSNDNCacheOp.java:[line 97] |
| Failed junit tests | hadoop.hdfs.server.namenode.TestRedudantBlocks |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15091 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12989762/HDFS-15091-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mv
[jira] [Updated] (HDFS-15079) RBF: Client maybe get an unexpected result with network anomaly
[ https://issues.apache.org/jira/browse/HDFS-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-15079: --- Attachment: HDFS-15079.001.patch > RBF: Client maybe get an unexpected result with network anomaly > > > Key: HDFS-15079 > URL: https://issues.apache.org/jira/browse/HDFS-15079 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Affects Versions: 3.3.0 >Reporter: Fei Hui >Priority: Critical > Attachments: HDFS-15079.001.patch, UnexpectedOverWriteUT.patch > > > I find there is a critical problem on RBF, HDFS-15078 can resolve it on some > Scenarios, but i have no idea about the overall resolution. > The problem is that > Client with RBF(r0, r1) create a file HDFS file via r0, it gets Exception and > failovers to r1 > r0 has been send create rpc to namenode(1st create) > Client create a HDFS file via r1(2nd create) > Client writes the HDFS file and close it finally(3rd close) > Maybe namenode receiving the rpc in order as follow > 2nd create > 3rd close > 1st create > And overwrite is true by default, this would make the file had been written > an empty file. This is an critical problem > We had encountered this problem. There are many hive and spark jobs running > on our cluster, sometimes it occurs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15079) RBF: Client maybe get an unexpected result with network anomaly
[ https://issues.apache.org/jira/browse/HDFS-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-15079: --- Attachment: (was: HDFS-15079.001.patch) > RBF: Client maybe get an unexpected result with network anomaly > > > Key: HDFS-15079 > URL: https://issues.apache.org/jira/browse/HDFS-15079 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Affects Versions: 3.3.0 >Reporter: Fei Hui >Priority: Critical > Attachments: UnexpectedOverWriteUT.patch > > > I find there is a critical problem on RBF, HDFS-15078 can resolve it on some > Scenarios, but i have no idea about the overall resolution. > The problem is that > Client with RBF(r0, r1) create a file HDFS file via r0, it gets Exception and > failovers to r1 > r0 has been send create rpc to namenode(1st create) > Client create a HDFS file via r1(2nd create) > Client writes the HDFS file and close it finally(3rd close) > Maybe namenode receiving the rpc in order as follow > 2nd create > 3rd close > 1st create > And overwrite is true by default, this would make the file had been written > an empty file. This is an critical problem > We had encountered this problem. There are many hive and spark jobs running > on our cluster, sometimes it occurs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15090) RBF: MountPoint Listing Should Return Flag Values Of Destination
[
https://issues.apache.org/jira/browse/HDFS-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006294#comment-17006294
]
Hadoop QA commented on HDFS-15090:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
8s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15090 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12989761/HDFS-15090-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 66fb7876ee45 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 074050c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28599/testReport/ |
| Max. process+thread count | 2797 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28599/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: MountPoint Listing Should Return Flag Values Of Destination
>
>
> Key:
[jira] [Updated] (HDFS-15091) Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock
[ https://issues.apache.org/jira/browse/HDFS-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15091: Status: Patch Available (was: Open) > Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock > > > Key: HDFS-15091 > URL: https://issues.apache.org/jira/browse/HDFS-15091 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15091-01.patch > > > As of now all API check superuser before taking lock, Similarly can be done > for the cache commands and setQuota. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15091) Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock
[ https://issues.apache.org/jira/browse/HDFS-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15091: Attachment: HDFS-15091-01.patch > Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock > > > Key: HDFS-15091 > URL: https://issues.apache.org/jira/browse/HDFS-15091 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15091-01.patch > > > As of now all API check superuser before taking lock, Similarly can be done > for the cache commands and setQuota. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15091) Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock
Ayush Saxena created HDFS-15091: --- Summary: Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock Key: HDFS-15091 URL: https://issues.apache.org/jira/browse/HDFS-15091 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ayush Saxena Assignee: Ayush Saxena As of now all API check superuser before taking lock, Similarly can be done for the cache commands and setQuota. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15090) RBF: MountPoint Listing Should Return Flag Values Of Destination
[ https://issues.apache.org/jira/browse/HDFS-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15090: Attachment: HDFS-15090-01.patch > RBF: MountPoint Listing Should Return Flag Values Of Destination > > > Key: HDFS-15090 > URL: https://issues.apache.org/jira/browse/HDFS-15090 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15090-01.patch > > > While doing listing, if a mount point is there and if the actual destination > is there, the owner and group details are taken from destination, similarly > the flags value can also be returned from destination. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15090) RBF: MountPoint Listing Should Return Flag Values Of Destination
[ https://issues.apache.org/jira/browse/HDFS-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15090: Status: Patch Available (was: Open) > RBF: MountPoint Listing Should Return Flag Values Of Destination > > > Key: HDFS-15090 > URL: https://issues.apache.org/jira/browse/HDFS-15090 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15090-01.patch > > > While doing listing, if a mount point is there and if the actual destination > is there, the owner and group details are taken from destination, similarly > the flags value can also be returned from destination. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15090) RBF: MountPoint Listing Should Return Flag Values Of Destination
Ayush Saxena created HDFS-15090: --- Summary: RBF: MountPoint Listing Should Return Flag Values Of Destination Key: HDFS-15090 URL: https://issues.apache.org/jira/browse/HDFS-15090 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ayush Saxena Assignee: Ayush Saxena While doing listing, if a mount point is there and if the actual destination is there, the owner and group details are taken from destination, similarly the flags value can also be returned from destination. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006285#comment-17006285 ] Hudson commented on HDFS-15063: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17802 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17802/]) HDFS-15063. HttpFS: getFileStatus doesn't return ecPolicy. Contributed (tasanuma: rev 074050ca595a81927c867951e48cef132a0284be) * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java > HttpFS : getFileStatus doesn't return ecPolicy > -- > > Key: HDFS-15063 > URL: https://issues.apache.org/jira/browse/HDFS-15063 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15063.001.patch, HDFS-15063.002.patch, > HDFS-15063.003.patch, HDFS-15063.004.patch, HDFS-15063.005.patch > > > Currently LISTSTATUS call to HttpFS returns a json. These jsonArray elements > have the ecPolicy name. > But when HttpFsFileSystem converts it back into a FileStatus object, then > ecPolicy is not added -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-15063: Fix Version/s: 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for your contribution, [~hemanthboyina]! > HttpFS : getFileStatus doesn't return ecPolicy > -- > > Key: HDFS-15063 > URL: https://issues.apache.org/jira/browse/HDFS-15063 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15063.001.patch, HDFS-15063.002.patch, > HDFS-15063.003.patch, HDFS-15063.004.patch, HDFS-15063.005.patch > > > Currently LISTSTATUS call to HttpFS returns a json. These jsonArray elements > have the ecPolicy name. > But when HttpFsFileSystem converts it back into a FileStatus object, then > ecPolicy is not added -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006280#comment-17006280 ] Takanobu Asanuma commented on HDFS-15063: - The failed tests are not related. I can fix the checkstyle warnings when committing it. +1. > HttpFS : getFileStatus doesn't return ecPolicy > -- > > Key: HDFS-15063 > URL: https://issues.apache.org/jira/browse/HDFS-15063 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15063.001.patch, HDFS-15063.002.patch, > HDFS-15063.003.patch, HDFS-15063.004.patch, HDFS-15063.005.patch > > > Currently LISTSTATUS call to HttpFS returns a json. These jsonArray elements > have the ecPolicy name. > But when HttpFsFileSystem converts it back into a FileStatus object, then > ecPolicy is not added -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15079) RBF: Client maybe get an unexpected result with network anomaly
[
https://issues.apache.org/jira/browse/HDFS-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006279#comment-17006279
]
Hadoop QA commented on HDFS-15079:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 1s{color}
| {color:red} root generated 1 new + 1868 unchanged - 0 fixed = 1869 total (was
1868) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 37s{color} | {color:orange} root: The patch generated 14 new + 268 unchanged
- 1 fixed = 282 total (was 269) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 30s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 45s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}236m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | org.apache.hadoop.ipc.RetryCache$CacheEntryContext overrides equals in
RetryCache$CacheEntry and may not be symmetric At RetryCache.java:and may not
be symmetric At RetryCache.java:[lines 198-205] |
| Failed junit tests | hadoop.security.TestFixKerberosTicketOrder |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.namenode.TestRedudantBlocks |
| | hadoop.hdfs.server.federation.router.TestRouterQuota |
| | hadoop.fs.contract.router.
[jira] [Updated] (HDFS-15079) RBF: Client maybe get an unexpected result with network anomaly
[ https://issues.apache.org/jira/browse/HDFS-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-15079: --- Attachment: HDFS-15079.001.patch > RBF: Client maybe get an unexpected result with network anomaly > > > Key: HDFS-15079 > URL: https://issues.apache.org/jira/browse/HDFS-15079 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Affects Versions: 3.3.0 >Reporter: Fei Hui >Priority: Critical > Attachments: HDFS-15079.001.patch, UnexpectedOverWriteUT.patch > > > I find there is a critical problem on RBF, HDFS-15078 can resolve it on some > Scenarios, but i have no idea about the overall resolution. > The problem is that > Client with RBF(r0, r1) create a file HDFS file via r0, it gets Exception and > failovers to r1 > r0 has been send create rpc to namenode(1st create) > Client create a HDFS file via r1(2nd create) > Client writes the HDFS file and close it finally(3rd close) > Maybe namenode receiving the rpc in order as follow > 2nd create > 3rd close > 1st create > And overwrite is true by default, this would make the file had been written > an empty file. This is an critical problem > We had encountered this problem. There are many hive and spark jobs running > on our cluster, sometimes it occurs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15079) RBF: Client maybe get an unexpected result with network anomaly
[ https://issues.apache.org/jira/browse/HDFS-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-15079: --- Attachment: (was: HDFS-15079.001.patch) > RBF: Client maybe get an unexpected result with network anomaly > > > Key: HDFS-15079 > URL: https://issues.apache.org/jira/browse/HDFS-15079 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Affects Versions: 3.3.0 >Reporter: Fei Hui >Priority: Critical > Attachments: UnexpectedOverWriteUT.patch > > > I find there is a critical problem on RBF, HDFS-15078 can resolve it on some > Scenarios, but i have no idea about the overall resolution. > The problem is that > Client with RBF(r0, r1) create a file HDFS file via r0, it gets Exception and > failovers to r1 > r0 has been send create rpc to namenode(1st create) > Client create a HDFS file via r1(2nd create) > Client writes the HDFS file and close it finally(3rd close) > Maybe namenode receiving the rpc in order as follow > 2nd create > 3rd close > 1st create > And overwrite is true by default, this would make the file had been written > an empty file. This is an critical problem > We had encountered this problem. There are many hive and spark jobs running > on our cluster, sometimes it occurs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[
https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006170#comment-17006170
]
Hadoop QA commented on HDFS-15063:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
101 unchanged - 0 fixed = 103 total (was 101) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
49s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
38s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}188m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.server.namenode.TestRedudantBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15063 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12989746/HDFS-15063.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 80f77382aef9 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | m
[jira] [Commented] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006102#comment-17006102 ] hemanthboyina commented on HDFS-15063: -- thanks for the review [~tasanuma] have updated the patch , please review > HttpFS : getFileStatus doesn't return ecPolicy > -- > > Key: HDFS-15063 > URL: https://issues.apache.org/jira/browse/HDFS-15063 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15063.001.patch, HDFS-15063.002.patch, > HDFS-15063.003.patch, HDFS-15063.004.patch, HDFS-15063.005.patch > > > Currently LISTSTATUS call to HttpFS returns a json. These jsonArray elements > have the ecPolicy name. > But when HttpFsFileSystem converts it back into a FileStatus object, then > ecPolicy is not added -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-15063: - Attachment: HDFS-15063.005.patch > HttpFS : getFileStatus doesn't return ecPolicy > -- > > Key: HDFS-15063 > URL: https://issues.apache.org/jira/browse/HDFS-15063 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15063.001.patch, HDFS-15063.002.patch, > HDFS-15063.003.patch, HDFS-15063.004.patch, HDFS-15063.005.patch > > > Currently LISTSTATUS call to HttpFS returns a json. These jsonArray elements > have the ecPolicy name. > But when HttpFsFileSystem converts it back into a FileStatus object, then > ecPolicy is not added -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14546) Document block placement policies
[ https://issues.apache.org/jira/browse/HDFS-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006058#comment-17006058 ] Amithsha edited comment on HDFS-14546 at 12/31/19 11:27 AM: Ya, we can add AvailableSpaceBlockPlacementPolicy, but I thought AvailableSpaceBlockPlacementPolicy is mostly on choosing volumes since it's not on node and rack. I have skipped this. was (Author: amithsha): Ya, we can add AvailableSpaceBlockPlacementPolicy, but I thought AvailableSpaceBlockPlacementPolicy is mostly on choosing volumes since it's not on node and rack. > Document block placement policies > - > > Key: HDFS-14546 > URL: https://issues.apache.org/jira/browse/HDFS-14546 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Amithsha >Priority: Major > Labels: documentation > Attachments: HDFS-14546-01.patch, HDFS-14546-02.patch, > HDFS-14546-03.patch, HDFS-14546-04.patch, HDFS-14546-05.patch, > HDFS-14546-06.patch, HDFS-14546-07.patch, HDFS-14546-08.patch, > HdfsDesign.patch > > > Currently, all the documentation refers to the default block placement policy. > However, over time there have been new policies: > * BlockPlacementPolicyRackFaultTolerant (HDFS-7891) > * BlockPlacementPolicyWithNodeGroup (HDFS-3601) > * BlockPlacementPolicyWithUpgradeDomain (HDFS-9006) > We should update the documentation to refer to them explaining their > particularities and probably how to setup each one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14546) Document block placement policies
[ https://issues.apache.org/jira/browse/HDFS-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006058#comment-17006058 ] Amithsha commented on HDFS-14546: - Ya, we can add AvailableSpaceBlockPlacementPolicy, but I thought AvailableSpaceBlockPlacementPolicy is mostly on choosing volumes since it's not on node and rack. > Document block placement policies > - > > Key: HDFS-14546 > URL: https://issues.apache.org/jira/browse/HDFS-14546 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Amithsha >Priority: Major > Labels: documentation > Attachments: HDFS-14546-01.patch, HDFS-14546-02.patch, > HDFS-14546-03.patch, HDFS-14546-04.patch, HDFS-14546-05.patch, > HDFS-14546-06.patch, HDFS-14546-07.patch, HDFS-14546-08.patch, > HdfsDesign.patch > > > Currently, all the documentation refers to the default block placement policy. > However, over time there have been new policies: > * BlockPlacementPolicyRackFaultTolerant (HDFS-7891) > * BlockPlacementPolicyWithNodeGroup (HDFS-3601) > * BlockPlacementPolicyWithUpgradeDomain (HDFS-9006) > We should update the documentation to refer to them explaining their > particularities and probably how to setup each one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15089) RBF: SmallFix for RBFMetrics in doc
[ https://issues.apache.org/jira/browse/HDFS-15089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15089: Priority: Trivial (was: Major) > RBF: SmallFix for RBFMetrics in doc > --- > > Key: HDFS-15089 > URL: https://issues.apache.org/jira/browse/HDFS-15089 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: luhuachao >Assignee: luhuachao >Priority: Trivial > > SmallFix for RBFMetrics in doc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15089) RBF: SmallFix for RBFMetrics in doc
luhuachao created HDFS-15089: Summary: RBF: SmallFix for RBFMetrics in doc Key: HDFS-15089 URL: https://issues.apache.org/jira/browse/HDFS-15089 Project: Hadoop HDFS Issue Type: Bug Reporter: luhuachao Assignee: luhuachao SmallFix for RBFMetrics in doc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006041#comment-17006041
]
Ayush Saxena commented on HDFS-15051:
-
Thanx [~hexiaoqiao] for the patch.
If the immediate parent doesn't exist, the parent above is checked for WRITE
permission only, IMO it should be EXECUTE only, If parent is there then we can
check WRITE, else we can cosider it exists virtually and has required
permissions, and move up normally.
Add some test coverage where the parent doesn't exist, immediate parent or one
of the parent or no parent exist.
[~elgoiri] give a check your use case stays secured and Is this compatible
change or not?
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch,
> HDFS-15051.003.patch, HDFS-15051.004.patch, HDFS-15051.005.patch,
> HDFS-15051.006.patch, HDFS-15051.007.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14546) Document block placement policies
[
https://issues.apache.org/jira/browse/HDFS-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006035#comment-17006035
]
Ayush Saxena commented on HDFS-14546:
-
Thanx [~Amithsha] for the update,
This misses {{AvailableSpaceBlockPlacementPolicy}}
Apart from that on a quick look, patch lgtm
> Document block placement policies
> -
>
> Key: HDFS-14546
> URL: https://issues.apache.org/jira/browse/HDFS-14546
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Amithsha
>Priority: Major
> Labels: documentation
> Attachments: HDFS-14546-01.patch, HDFS-14546-02.patch,
> HDFS-14546-03.patch, HDFS-14546-04.patch, HDFS-14546-05.patch,
> HDFS-14546-06.patch, HDFS-14546-07.patch, HDFS-14546-08.patch,
> HdfsDesign.patch
>
>
> Currently, all the documentation refers to the default block placement policy.
> However, over time there have been new policies:
> * BlockPlacementPolicyRackFaultTolerant (HDFS-7891)
> * BlockPlacementPolicyWithNodeGroup (HDFS-3601)
> * BlockPlacementPolicyWithUpgradeDomain (HDFS-9006)
> We should update the documentation to refer to them explaining their
> particularities and probably how to setup each one of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15087) RBF: Balance/Rename across federation namespaces
[
https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006033#comment-17006033
]
Ayush Saxena commented on HDFS-15087:
-
Thanx [~LiJinglun] for the initiative.
Just had a quick skim through the idea, This majorily seems to revolve around
hard links for copy. You can give a check to HDFS-2139, ({{FastCopy}}), You may
take a look at the approach there, Remember trying that internally long back
and IIRC the Hard Links doesn't work in windows, but it had some logic by which
copy used to be success in windows too. Since I remember my UT for {{FastCopy}}
used to pass in windows too.
Give a check to HDFS-2139, if it is still valid, we can conclude that and call
{{FastCopy}} directly here.
> RBF: Balance/Rename across federation namespaces
>
>
> Key: HDFS-15087
> URL: https://issues.apache.org/jira/browse/HDFS-15087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jinglun
>Priority: Major
> Attachments: HFR_Rename Across Federation Namespaces.pdf
>
>
> The Xiaomi storage team has developed a new feature called HFR(HDFS
> Federation Rename) that enables us to do balance/rename across federation
> namespaces. The idea is to first move the meta to the dst NameNode and then
> link all the replicas. It has been working in our largest production cluster
> for 2 months. We use it to balance the namespaces. It turns out HFR is fast
> and flexible. The detail could be found in the design doc.
> Looking forward to a lively discussion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
