[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288235#comment-17288235
]
zhengchenyu commented on HDFS-14343:
[~ayushtkn] Maybe I think we should discuss in another issue HDFS-15750.
I known there is no problem if all hdfs client visit namenode through router.
But I think there are some situation that visit namnode without router.
For example, we migrating data from one nameservice to another nameservice in
background, and two nameservie are managed by same router, we migration data to
lower the pressure of source namenode.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0, HDFS-13891
>
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274357#comment-17274357
]
Ayush Saxena commented on HDFS-14343:
-
Hi [~zhengchenyu],
For your case-1 :
bq. If /user/userA is mountable which mounts two nameservice: ns1, ns2. But if
both hdfs://ns1/user/userA/a.log and hdfs://ns2/user/userA/a.log exists. I want
to remove hdfs://ns-fed/user/userA/a.log (Note: a.log is file) to trash, then
only one nameservice take effect.
In multiple destination scenarios, same Directories can be there in both
namespaces, but not files, Try creating a file through router it won't allow
you to create in another namespace, if the same path exists in one namespace.
We have a check during the create call in case there are multiple destinations,
check if the file already exists.
Such a scenario can only happen if someone directly goes to the namenode and
create the file instead of going through Router, And that isn't a valid use
case, If you create through router you would be able to delete through it
without any issues.
case-2
bq. In other way, if we hdfs://ns-fed/user/userA/dirA (Note: dirA is
directroy.) If hdfs://ns1/user/userA/dirA's permission is not same with
hdfs://ns2/user/userA/dirA's permission.
Again if you set permission through router, permissions would be same in both
namespaces,
Case-3
In case one nameservice is down, the call will fail unless fault tolerance
isn't turned on. That is another broad subject...
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0, HDFS-13891
>
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254391#comment-17254391
]
zhengchenyu commented on HDFS-14343:
[~elgoiri] OK, Let us discuss this issue in HDFS-15750. I will describe the
detailed later.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0, HDFS-13891
>
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254162#comment-17254162
]
Íñigo Goiri commented on HDFS-14343:
The multi destination semantics are a little special so there is a chance for
behaviors like the one you describe.
Can you open a JIRA covering this case and start with a unit test that does
what you describe?
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0, HDFS-13891
>
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253343#comment-17253343
]
zhengchenyu commented on HDFS-14343:
[~inigoiri][~ayushtkn].
Hi, I have some question, can you give me some suggestion. In this patch, use
isMultiDestDirectory to check whether rename all location or not. I think the
design of MultipleDestinationMountTableResolver may consider there is no
repeated file among multi nameservice.
If /user/userA is mountable which mounts two nameservice: ns1, ns2. But if both
hdfs://ns1/user/userA/a.log and hdfs://ns2/user/userA/a.log exists. I want to
remove hdfs://ns-fed/user/userA/a.log (Note: a.log is file) to trash, then only
one nameservice take effect. I think it means inconsistence.
(Note: In fact, when migration data from one nameservice to nameservice, and
rewrite some hive table's old partition, this problem would occure!)
In other way, if we hdfs://ns-fed/user/userA/dirA (Note: dirA is directroy.) If
hdfs://ns1/user/userA/dirA's permission is not same with
hdfs://ns2/user/userA/dirA's permission. Or one of nameservice is down. We
maybe can't rename all location. I think it also means inconsistence. I think
we need stricter check. If one operation error, we should throw exception.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0, HDFS-13891
>
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871610#comment-16871610
]
Hudson commented on HDFS-14343:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16813 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16813/])
HDFS-14343. RBF: Fix renaming folders spread across multiple (brahma: rev
f539e2a4ee93c4ee479fe25e8062c8ab4c7f8ba8)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRPCMultipleDestinationMountTableResolver.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788238#comment-16788238
]
Giovanni Matteo Fumarola commented on HDFS-14343:
-
Thanks [~ayushtkn] for working on this and [~elgoiri] for the review.
LGTM +1.
Committed to the branch.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788152#comment-16788152
]
Íñigo Goiri commented on HDFS-14343:
Thanks [~ayushtkn] for [^HDFS-14343-HDFS-13891-05.patch].
+1
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787930#comment-16787930
]
Hadoop QA commented on HDFS-14343:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
51s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
57s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14343 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12961723/HDFS-14343-HDFS-13891-05.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f561fac89cd2 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bddb4cb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26437/testReport/ |
| Max. process+thread count | 1000 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26437/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix renaming folders spread across multiple subclusters
>
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787871#comment-16787871
]
Ayush Saxena commented on HDFS-14343:
-
uploaded v5 removing the break line
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch, HDFS-14343-HDFS-13891-05.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787556#comment-16787556
]
Hadoop QA commented on HDFS-14343:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
49s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 28m
27s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14343 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12961657/HDFS-14343-HDFS-13891-04.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 16e2aece4489 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bddb4cb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26434/testReport/ |
| Max. process+thread count | 1369 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26434/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix renaming folders spread across multiple subclusters
>
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787529#comment-16787529
]
Íñigo Goiri commented on HDFS-14343:
Just a minor comment: avoid break line in RouterClientProtocol#1892.
Other than that this LGTM.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787487#comment-16787487
]
Ayush Saxena commented on HDFS-14343:
-
Thanx [~elgoiri] for the comments.
Addressed as part of v4.
Pls Review :)
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch,
> HDFS-14343-HDFS-13891-04.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787490#comment-16787490
]
Hadoop QA commented on HDFS-14343:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
56s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
5s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14343 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12961647/HDFS-14343-HDFS-13891-03.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a574cd65936e 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bddb4cb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26431/testReport/ |
| Max. process+thread count | 1019 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26431/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix renaming folders spread across multiple subclusters
>
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787479#comment-16787479
]
Íñigo Goiri commented on HDFS-14343:
Thanks [~ayushtkn] for [^HDFS-14343-HDFS-13891-03.patch].
* I think we can make {{isMultiDestDirectory()}} to be package visible (no
modified) instead of public.
* Let's just make the log in line 1887 a debug log, this is a normal code path,
no need for warning.
* Don't know what to do with the {{UnresolvedPathException}}, is it worth a
warning or debug is enough?
* Instead of log an throw for the IOException, let's just let it go throw
without catching and logging. The receiver of the exception will log if needed.
* The unit tests looks good. We may want to define some of the variables when
we need them. For example, dst2 and nnDst2 can be defined in line 532.
* As {{testRenameMultipleDestDirectories()}} and
{{verifyRenameOnMultiDestDirectories()}} are directly connected, we may want to
do {{testIsMultiDestDir()}} before both.
Other than this minor comments, I think this is pretty much ready.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787437#comment-16787437
]
Ayush Saxena commented on HDFS-14343:
-
Uploaded v3 addressing comments.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch, HDFS-14343-HDFS-13891-03.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787134#comment-16787134
]
Íñigo Goiri commented on HDFS-14343:
{quote}
For a file that is not present. The file status returned is null rather than
giving a FileNotFoundException. So we can have a null check at
RouterClientProtocol#1880
{quote}
Perfect, let's just constrain the exceptions we expect and log error for the
generic exception.
Maybe even throw those and let them surface.
{quote}
Regarding the symlink. I guess getFileInfo isn't behaving as we want. For
symLink instead returning the status. It throws back exception telling it is
symlink, It has a check for symLink at namenode side. So I guess for this we
need to catch exception. If I am ain't missing something.
{quote}
Yes, let's catch {{UnresolvedPathException}}.
The current unit test should have good coverage for this.
{quote}
I didn't catch this. Do you mean checking if there is file in all subclusters?
If so, if we invoke RPC at all subcluster. we would land back to the stage
worse than directly calling invokeAll after checking isPathAll() in terms of
RPC calls.(since now we are having an overhead of calculation too).
{quote}
I was just referring to checking for files within the directories to make sure
they were in the src and then they are in the dst too.
It wouldn't hurt to add the rename of a file too (even though it is covered by
TestRouteRpc*); it would be just one line in the unit test.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787113#comment-16787113
]
Hadoop QA commented on HDFS-14343:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
8s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
42s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14343 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12961599/HDFS-14343-HDFS-13891-02.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fda4680accd3 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bddb4cb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26428/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26428/testReport/ |
| Max. process+thread count | 994 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787083#comment-16787083
]
Ayush Saxena commented on HDFS-14343:
-
For a file that is not present. The file status returned is null rather than
giving a FileNotFoundException. So we can have a null check at
{{RouterClientProtocol#1880}}
Regarding the symlink. I guess getFileInfo isn't behaving as we want. For
symLink instead returning the status. It throws back exception telling it is
symlink, It has a check for symLink at namenode side. So I guess for this we
need to catch exception. If I am ain't missing something.
{noformat}
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnresolvedPathException):
/tmp/dir/file
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkNotSymlink(FSPermissionChecker.java:657)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:616)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1823)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1841)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:681)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:109)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3205)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1197){noformat}
{quote}For example verify that the files are in both time.
{quote}
I didn't catch this. Do you mean checking if there is file in all subclusters?
If so, if we invoke RPC at all subcluster. we would land back to the stage
worse than directly calling invokeAll after checking isPathAll() in terms of
RPC calls.(since now we are having an overhead of calculation too).
Pls correct if going somewhere wrong.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787034#comment-16787034
]
Íñigo Goiri commented on HDFS-14343:
I forgot: can we add checks for files in verifyRenameOnMultiDestDirectories()?
For example verify that the files are in both time.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787029#comment-16787029
]
Íñigo Goiri commented on HDFS-14343:
* I guess {{TestRouterRPCMultipleDestinationMountTableResolver#455}} should be
a file path (there is two nodir).
* What about links to directories?
* Is there a chance {{RouterClientProtocol#1880}} can return null?
* We may want to capture the expected exception (FileNotFound I guess) in
{{RouterClientProtocol#1884}}. Then we can capture the general exception but
log it as an error. I think we may have other exceptions like
NamenodeNotAvailable and other RBF exceptions too. Actually, we may want to let
those surface.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787016#comment-16787016
]
Ayush Saxena commented on HDFS-14343:
-
Added v2 addressing the comments.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch,
> HDFS-14343-HDFS-13891-02.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786966#comment-16786966
]
Hadoop QA commented on HDFS-14343:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
57s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m
37s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14343 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12961579/HDFS-14343-HDFS-13891-01.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2ab74017a54c 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / bddb4cb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26426/testReport/ |
| Max. process+thread count | 1013 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26426/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix renaming folders spread across multiple subclusters
>
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786964#comment-16786964
]
Íñigo Goiri commented on HDFS-14343:
I went a little deeper and in addition to the previous comments:
* Restrict the visibility of {{isMultiDestDirectory()}} and mark it as
VisibleForTesting for the tests I mentioned before.
* Handle error cases better in {{isMultiDestDirectory()}} like file not found,
null outputs etc.
* For the test I would put all the variations together in a single test
function.
* We should test some corner cases like one of the namespaces not having the
source folder and one of the destinations already existing.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786929#comment-16786929
]
Íñigo Goiri commented on HDFS-14343:
Thanks [~ayushtkn] for the patch.
I need to take a closer look but I think we should add a unit test for
isMultiDestDirectory().
We should cover:
* Files
* Directories
* Non existing
* Weird permissions like a directories we don't have access to.
* Others like links?
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786864#comment-16786864
]
Ayush Saxena commented on HDFS-14343:
-
Have uploaded path v1 with the fix.
Pls Review!!!
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14343-HDFS-13891-01.patch
>
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786433#comment-16786433
]
Íñigo Goiri commented on HDFS-14343:
{quote}
Íñigo Goiri I guess this shall be somewhere near HDFS-14226.
Can take this up. if no issues
{quote}
Awesome, this should be pretty similar.
Maybe some complexity distinguishing files from folders but doable.
The unit test should be pretty simple.
I would also like to add some cases where the folder is missing in some
subcluster or similar.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Priority: Major
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786420#comment-16786420
]
Ayush Saxena commented on HDFS-14343:
-
[~elgoiri] I guess this shall be somewhere near
[HDFS-14226.|https://github.com/apache/hadoop/commit/9b1c43ec42cbbccbbd1e4b2f765ccd2bdb53baac#diff-797a75e60d13fac869521c26440a8efa]
Can take this up. if no issues :)
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Priority: Major
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14343) RBF: Fix renaming folders spread across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786414#comment-16786414
]
Íñigo Goiri commented on HDFS-14343:
If nobody is interested on working on this, I will start working on it tomorrow.
> RBF: Fix renaming folders spread across multiple subclusters
>
>
> Key: HDFS-14343
> URL: https://issues.apache.org/jira/browse/HDFS-14343
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Priority: Major
>
> The {{RouterClientProtocol#rename()}} function assumes that we are renaming
> files and only renames one of them (i.e., {{invokeSequential()}}). In the
> case of folders which are in all subclusters (e.g., HASH_ALL) we should
> rename all locations (i.e., {{invokeAll()}}).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
