[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17600659#comment-17600659 ] zhengchenyu commented on HDFS-14117: In our cluster, I must mount all nameservice for /user/${user}/.Trash, it means router will rename all nameservice when move to trash. Though it works for long time, this will result to bad performance when one namenode degrade. I wanna only connect one nameservice. So I have a new proposal: Condition: (1) /test is mounted in ns0 (2) /user/hdfs is mounted is ns1 If we move /test/hello to /user/hdfs/.Trash/Current/test/hello. When we process the location with trash prefix, we just use the location which remove the prefix to find the mounted ns. For /user/hdfs/.Trash/Current/test/hello, we remove the prefix '/user/hdfs/.Trash/Current', get '/test/hello', use '/test/hello' to find the mounted ns. Then we got the location: ns0->/user/hdfs/.Trash/Current/test/hello, then rename to trash will work. The problem is that we must check the pattern of location in every call, but I think it is low cost. [~elgoiri] [~ayushtkn] [~hexiaoqiao] [~ramkumar] [~xuzq_zander] How about my proposal? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ramkumar Ramalingam >Assignee: Ramkumar Ramalingam >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: >
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969522#comment-16969522 ] Ayush Saxena commented on HDFS-14117: - Well, from my side too, I don't have any solution. :( I tried on many solutions, but all seems to have some loop hole, Couldn't find a perfect one. [~hexiaoqiao] Let us know if you found one. Not for rename but if we just focus trash and try to be smart at router as already opposed. we can atleast get the trash support going provided we have /user mount entry pointing to namespace user entry. The need for having /user mount also can also be eliminated, not very good way but if we want we can by internally resolving the path to respective trash path and invoke to respective namespace if to TRASH is there. But we need to do quite a similar resolution for read request too, That might not be easy and trying to do, if not done very carefully may open up some security loop holes, and may allow a hacky user to access the namespace. Need to be quite sure on this. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ramkumar Ramalingam >Assignee: Ramkumar Ramalingam >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969509#comment-16969509 ] Íñigo Goiri commented on HDFS-14117: Not from my side. I think at this point we should have a JIRA to just trigger a proper message when this happens. Then we can work on this. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ramkumar Ramalingam >Assignee: Ramkumar Ramalingam >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969039#comment-16969039 ] Xiaoqiao He commented on HDFS-14117: Hi [~ramkumar],[~elgoiri],[~ayushtkn], any progress here? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ramkumar Ramalingam >Assignee: Ramkumar Ramalingam >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930728#comment-16930728 ] Íñigo Goiri commented on HDFS-14117: The main issue here is not only move to trash but renaming in general. I started working on HDFS-14431 but this has a lot of corner cases that can get pretty expensive. We need to resolve that one first. If anybody is up to take it, please go ahead. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930642#comment-16930642 ] Ayush Saxena commented on HDFS-14117: - Daryn’s comment needs to be addressed, as of now we don’t have a solution to handle without checking To Trash as opposed by Daryn. And one more is that, the present solution is dependent on availability of /user mount entry pointing to all Ns /user [~elgoiri] I think is working on this.. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930422#comment-16930422 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color} | {color:red} HDFS-14117 does not apply to HDFS-13891. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12966115/HDFS-14117-HDFS-13891.020.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/27884/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003) - To
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930414#comment-16930414 ] Yuxuan Wang commented on HDFS-14117: Hi [~elgoiri], [~xuzq_zander], [~ayushtkn]. Are you still working on this? I think the patch is critical for clusters running in real world. After go through the related jira, I can't figure out where we block? I hope someone can remind me thx. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851678#comment-16851678 ] xuzq commented on HDFS-14117: - Like this: * /home/hdfs/dir1 mount to ns1 * /home/hdfs/dir2 mount to ns2 * /home/hdfs/dir3 mount to ns3 * /user/hdfs mount to ns1 when we delete /home/hdfs/dir3/file3, it will be automatically moved to /user/hdfs/.Trash/Current/home/hdfs/dir3/file3 in ns3. It don't focus on the real mount point of /user/hdfs. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851107#comment-16851107 ] Íñigo Goiri commented on HDFS-14117: What's the exact behavior in ViewFs? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850843#comment-16850843 ] xuzq commented on HDFS-14117: - Thanx [~ramkumar] for this patch, good jobs(y). Look the test case in TestRouterTrash.class, we still need to mount the /user to the ns? If we add a new ns for the /home/hdfs from ns1:ns2 to ns1:ns2:ns3, we have to add the new ns to /user/hdfs:(. Can we rm the data like the ViewFs? doesn't care the /user/xxx mount point. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819649#comment-16819649 ] Íñigo Goiri commented on HDFS-14117: I created HDFS-14431 to do the fixes for renaming. Once that's in, we can discuss what to do with the Trash. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819403#comment-16819403 ] Íñigo Goiri commented on HDFS-14117: Thanks [~daryn] for the feedback. There are two parts to this JIRA. The first one is to improve the way rename works by taking into account incompatible subclusters. The second one is the trash itself. We can do the first one in a separate JIRA as it is the more fundamental one. The second one, it's isolated to the createTrash function. I tend to agree that the ideal scenario would be to provision the Trash before any of this. However, in default setups, this will fail. Maybe make the creation optional? Note that this is all based on the TRASH flag and there are no hard-coded paths. RBF does not support EZ at this point so I haven't been able to test that. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819378#comment-16819378 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 5s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 27s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 11s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 74m 33s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.fs.contract.router.TestRouterHDFSContractRename | | | hadoop.fs.contract.router.TestRouterHDFSContractRenameSecure | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12966115/HDFS-14117-HDFS-13891.020.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux fb4d86d5d399 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / bd3161e | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26648/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26648/testReport/ | | Max. process+thread count | 1004 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819336#comment-16819336 ] Daryn Sharp commented on HDFS-14117: It's a big patch and caught my eye because we are adding support for an alternate hdfs trash path and this will of course complicate everything. Anyway, the RBF is attempting to be too smart. It should know nothing about the trash. The predecessor to httpfs (hdfsproxy) had all kinds of issues because it tried to be smart instead of proxying the call through as-is. Very subtle race conditions cropped up that were hard to debug. Why not require the overlapping mount points to have a trash dir and then do the fan-out rename? Is this going to play nicely with EZ trashes? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819277#comment-16819277 ] Íñigo Goiri commented on HDFS-14117: Thanks [~ayushtkn] for the feedback. I posted [^HDFS-14117-HDFS-13891.020.patch] with the approach that checks all files. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819155#comment-16819155 ] Ayush Saxena commented on HDFS-14117: - Thanx [~elgoiri] for the briefings. Tough choice to make, when both solutions are technically correct. BEST vs BEST leads to one and WORST vs WORST leads to another. Some point I feel that Second case hitting the best case may be more but as we can't guarantee it, so we need to consider an average case considering both positive and negative cases happen at almost same rate, So I won't make a call based on number of RPC here. Preventing writeLock shall be then a better choice vs saving RPC's. If you too are good with it, I am happy going ahead with it. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818375#comment-16818375 ] Hadoop QA commented on HDFS-14117: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 45s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 36s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 53s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m 24s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 52s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965986/HDFS-14117-HDFS-13891.019.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 5ffb11b32f98 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26636/testReport/ | | Max. process+thread count | 1355 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26636/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818190#comment-16818190 ] Íñigo Goiri commented on HDFS-14117: The failed unit tests is caused by not doing the check for the file locations. The issue is that now we don't know where the file really is (we just now it's in the federation). Then we know that the file might be in ns0 or ns1 and we know the destination is in ns1. When we trigger the rename, we only trigger it against ns1. Then we get a FileNotFoundException from ns1 when it should be a "not eligible location". The way to handle this would be to infer that if we found the file first and now we get a FileNotFoundException, it must be because the src/dst is not allowed. With the previous approach we knew this failure case in just the {{getFileInfo}} while now we have to do a {{getFileInfo}} and a {{rename2}} to realize. Here there is a trade off between the amount of calls in the success and failure code paths and readability. To summarize: * First approach ([^HDFS-14117-HDFS-13891.017.patch]): read everywhere and rename in the right place. ** Positive case: getFileInfo in all subclusters, rename2 in the right subcluster. ** Negative case: getFileInfo in all subclusters. No rename2 needed. * Second approach ([^HDFS-14117-HDFS-13891.018.patch]); read in order and rename everywhere. ** Positive case: getFileInfo in the right subcluster, rename2 in the right subcluster. ** Negative case: getFileInfo in the right subcluster, rename2 in all subclusters (+ a non-intuitive code path that catches a FileNotFoundException and infers a NotEligibleRenameException). Keep in mind rename2 uses the writeLock while getFileInfo only readLock. I'm fine with either one. [~ayushtkn] which one you prefer? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117-HDFS-13891.018.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817553#comment-16817553 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 18s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 5s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 23s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 46s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 80m 13s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterTrash | | | hadoop.hdfs.server.federation.router.TestRouterFaultTolerant | | | hadoop.hdfs.server.federation.router.TestRouterRpc | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965898/HDFS-14117-HDFS-13891.018.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 62cc7e27ffbf 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26632/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26632/testReport/ | | Max. process+thread count | 1006 (vs. ulimit of 1) | | modules | C:
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817511#comment-16817511 ] Íñigo Goiri commented on HDFS-14117: The main issue here is {{RenameParam#checkSrcDst()}}. This comes from the old {{RouterClientProtocol#getRenameDestinations()}} Now we avoid sending the requests to the subclusters that don't have the file/folder. I guess I can be convinced that this optimization is not that useful and it is OK to just trust the first subcluster with the file/dir. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817177#comment-16817177 ] Ayush Saxena commented on HDFS-14117: - The reason with which we started in HDFS-14343 to check directories is to save the RPC by directly doing invokeConcurrent. I guess that goes away now. If we don't check and directly do an invokeConcurrent for isPathAll() that should be cheaper than doing this. In a ideal case of directories(which shall be true if everything is good) directories shall be in all the subclusters, so we shall doing twice the number of RPC calls, for a file in general two calls would have been enough, one to check for if it is a file and second for the operation, here also we are increasing the number of RPC, I am not sure with this whether this is correct thing to do. I guess we have increased the cost for non isPathAll() too? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817171#comment-16817171 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 39s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 33s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 11s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 44s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 27s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 86m 35s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.fs.contract.router.TestRouterHDFSContractRename | | | hadoop.fs.contract.router.TestRouterHDFSContractRenameSecure | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965832/HDFS-14117-HDFS-13891.017.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e89d600bac1c 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26629/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26629/testReport/ | | Max. process+thread count | 1051 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817155#comment-16817155 ] Íñigo Goiri commented on HDFS-14117: Thanks [~ayushtkn] for the comments. The reason for checking in all the src subclusters is to check for possible inconsistency issues. Assuming best cases we can reduce some calls but I think is safer to check everywhere and then send to the right locations. I added [^HDFS-14117-HDFS-13891.017.patch] with some refactoring. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817005#comment-16817005 ] Hadoop QA commented on HDFS-14117: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 19s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 11s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m 50s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 87m 57s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965812/HDFS-14117-HDFS-13891.016.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 109d4acad94a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26626/testReport/ | | Max. process+thread count | 1076 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26626/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816973#comment-16816973 ] Ayush Saxena commented on HDFS-14117: - Thanx [~elgoiri] for the patch. Overall I guess its almost done. To me it doesn't appear any loss to functionality, I guess all should stay intact. There is a test failure, I guess since you have hard-coded the user name, I replaced it with {{System.getProperty("user.name")}} and it works. Give a check, if that works or something similar. One doubt too regarding this: {code:java} Map srcFiles = rpcClient.invokeConcurrent( srcLocs, method, false, false, -1, HdfsFileStatus.class); // Check if the src exists and is a directory{code} Is this just to check existence of file and is it a directory? If so we can use invokeSequential and save rpc. In ideal case if its a directory it should be there in all and file shall always be in one. {code:java} for (final HdfsFileStatus srcFile : srcFiles.values()) { if (srcFile != null) { found = true;{code} Why don't we break after the file is found? file shall be one only I guess. In TestRouterRPC and TestRouterRPCMultiDestination in the test, we can directly use routerFS rather than re calculating or getRouterFileSystem() > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} --
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816819#comment-16816819 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 17s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 10s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 57s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 19s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 7s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 26s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterTrash | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965780/HDFS-14117-HDFS-13891.015.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 13996eb99c45 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26624/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26624/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816747#comment-16816747 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 13s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 2s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 54s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 4 new + 6 unchanged - 0 fixed = 10 total (was 6) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 7s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 51s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 71m 53s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.fs.contract.router.TestRouterHDFSContractRenameSecure | | | hadoop.fs.contract.router.TestRouterHDFSContractRename | | | hadoop.hdfs.server.federation.router.TestRouterTrash | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965767/HDFS-14117-HDFS-13891.014.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 91fadf5ce3b8 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26622/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | unit |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816740#comment-16816740 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 33s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 43s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 14s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 5 new + 5 unchanged - 0 fixed = 10 total (was 5) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 9s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 20s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 79m 39s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.fs.contract.router.TestRouterHDFSContractRename | | | hadoop.hdfs.server.federation.router.TestRouterTrash | | | hadoop.fs.contract.router.TestRouterHDFSContractRenameSecure | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965765/HDFS-14117-HDFS-13891.013.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9121f92bc256 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26620/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | unit |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816712#comment-16816712 ] Íñigo Goiri commented on HDFS-14117: I did {{isMultiDestDirectory()}} within the rest of the renaming and now is hard to have the all method. Take a look and check if we are losing functionality or coverage now. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117-HDFS-13891.014.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816691#comment-16816691 ] Íñigo Goiri commented on HDFS-14117: [^HDFS-14117-HDFS-13891.013.patch] should cover everything. It also goes some over the HDFS-14343 things that were left. For example, now we have a better control for the src/dst eligibility. I think {{isMultiDestDirectory()}} can be refactor to use some of the file info we get before this method. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815063#comment-16815063 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 38s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 20s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 48s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 5s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 85m 34s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterTrash | | | hadoop.hdfs.server.federation.router.TestRouterRpc | | | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination | | | hadoop.hdfs.server.federation.router.TestRouterRpcSingleNS | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965540/HDFS-14117-HDFS-13891.012.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f9d8ca4ddf25 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26613/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | unit |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815025#comment-16815025 ] Íñigo Goiri commented on HDFS-14117: Thanks for the comments [~ayushtkn]. I added more tests and I found a couple issues. There are still some things to figure. For example, the way the error is reported is not very clean and it should report the proper issue. This last point relates to mounting {{/user}}. Let's start with the case where we have {{/mnt}} mounted on {{ns0}} and {{ns1}} mounted and then we have files in each. / by default is mounted only to {{ns0}}. If we try to remove a file that is in {{ns1}} we cannot move the file from {{ns1}} to the Trash in {{ns0}}. For us to be able to move to the Trash, we need {{/user/user0/.Trash}} to be available in {{ns1}}. For this to have we have to have a mount point for {{/}} or {{/user}} or {{/user/user0}} or {{/user/user0/.Trash}} with a destination subcluster {{ns1}} I'll fix the rest of the comments once I figure how to provide a better exception. Thanks [~crh] for taking a look, let's follow up on the other JIRA. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117-HDFS-13891.012.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815018#comment-16815018 ] CR Hota commented on HDFS-14117: [~elgoiri] Thanks for reporting the test failure. I tried it reproduce it locally and could not. However upon multiple executions of another TestRouterHttpDelegationToken I did reproduce this once locally. {code:java} [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.076 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken [ERROR] testGetDelegationToken(org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken) Time elapsed: 0.115 s <<< ERROR! org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: router/localh...@example.com from keytab /hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab javax.security.auth.login.LoginException: Integrity check on decrypted field failed (31) - PREAUTH_FAILED at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173) at org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken.setup(TestRouterHttpDelegationToken.java:99) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: router/localh...@example.com from keytab /hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab javax.security.auth.login.LoginException: Integrity check on decrypted field failed (31) - PREAUTH_FAILED at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1376) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1156) at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:315) at org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:159) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) ... 27 more {code} Looks to me like some intermittent issue with the way
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814007#comment-16814007 ] Ayush Saxena commented on HDFS-14117: - Thanx [~elgoiri] for taking over!!! * For the trash to work, the availability of /user is mandatory? {code:java} +// Destination folder for the Trash to succeed +createMountTableEntry(rc, "/user", DestinationOrder.HASH); {code} {quote}What we internally do is to mount the .Trash folder (e.g., /user/hadoop/.Trash). It's a little hacky to set it up this way but it should work. {quote} Isn't it little similar to the approach you told earlier? Would we be fetching any advantage after the code change? For trash to work the admin has to mount one entry only? why not mount the one you suggested. Well might work if admin already had /user for some purpose(Maybe if possible, Let me know if any other ways this can help or if I messed up):) * The tests look pretty good, we have covered the working case for files in hash order, Can we have coverage for one Dir_All order and directories too? * Saw the {{createMountTableEntry(..)}} method at couple of other places too. Can you give a check can we move this to some test util and use it directly rather than having it at all places? * Regarding the mock Namenode that makes sense, We are having multiple namespaces, I feel that would surely shed some considerable time in the UT. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814003#comment-16814003 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 52s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 17s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 54s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 14s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 26s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 33s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 20s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 31s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965400/HDFS-14117-HDFS-13891.011.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3b56a826360b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/PreCommit-HDFS-Build/26605/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | compile | https://builds.apache.org/job/PreCommit-HDFS-Build/26605/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | javac |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813978#comment-16813978 ] Íñigo Goiri commented on HDFS-14117: Thanks for the comments [~ayushtkn]. Added a test for when the Trash is properly setup for multiple subclusters and when is missing. This should also cover the full use of the Trash. I was thinking on using the MockNamenode; let me know if needed. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813970#comment-16813970 ] Hadoop QA commented on HDFS-14117: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 47s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 48s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 43s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 26s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m 20s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}111m 4s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965386/HDFS-14117-HDFS-13891.010.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 143e47f9e7dc 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26603/testReport/ | | Max. process+thread count | 996 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26603/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813918#comment-16813918 ] Ayush Saxena commented on HDFS-14117: - Thanx [~elgoiri] for the patch. I guess as part we are covering the rename API with the Trash param. Can we extend a test covering the scenario that the JIRA talks. The delete scenario with trash enabled with multi dests. I doubt, there might be some dependency on availability of a /user mount or / root mount in the actual scenario. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813912#comment-16813912 ] Íñigo Goiri commented on HDFS-14117: I uploaded [^HDFS-14117-HDFS-13891.010.patch] fixing the checkstyles. [~crh], there is an error regarding the secure start of the Router but I cannot see how this is related; any idea? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117-HDFS-13891.010.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813842#comment-16813842 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 19s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 55s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 57s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 13s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 10s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 17s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 73m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12965366/HDFS-14117-HDFS-13891.009.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux be665cd82f61 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / e508ab9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26602/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26602/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813731#comment-16813731 ] Íñigo Goiri commented on HDFS-14117: I added [^HDFS-14117-HDFS-13891.009.patch] targetting my comments. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802621#comment-16802621 ] Íñigo Goiri commented on HDFS-14117: I think we can also add a rule in the {{HashResolver}} to transform {{/user/user0/.Trash/path/to/file}} into {{/path/to/file}}. [~ramkumar] can you take care of the comments? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762357#comment-16762357 ] venkata ramkumar commented on HDFS-14117: - Hi [~elgoiri] Thanks for the continuous support. Is this fix is correct functionally or not ? Since the comments seems to be non functional changes . If you are fine with the fix functionally then i can proceed further with the above comments. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761969#comment-16761969 ] Íñigo Goiri commented on HDFS-14117: * The log message in RouterClientProtocol#510, it should be report {{destLoc}} and not {{locations}}. Actually, shouldn't we be adding to the list? It looks like we will end up creating the Trash in just one subcluster. * Instead of breaking in RouterClientProtocol#521, we should show each of the errors; actually, we should report the failed location and not just locations. Let's use {{entrySet()}} for that. * There is no use for the {{success}} variable in {{RouterClientProtocol#515}} * The javadoc in {{testTrashSingleNameSpace()}} should start with capitals. * We probably want to put {{TestRouterRpc#testTrash}} and {{TestRouterRpc#testTrashSingleNameSpace()}} close to each other. * Add a javadoc to {{TestRouterRpc#testTrash}}. Mainly for the parameters. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761693#comment-16761693 ] venkata ramkumar commented on HDFS-14117: - Hi [~elgoiri] Uploaded the patch based on review comments.Please review it. * Test case failures are not related to this patch. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117-HDFS-13891.008.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761663#comment-16761663 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 31s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 52s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 52s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 91m 13s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.federation.store.driver.TestStateStoreZK | | | hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefresh | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12957737/HDFS-14117-HDFS-13891.008.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8a82500ede31 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / ecd90a6 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26161/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26161/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761541#comment-16761541 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} HDFS-14117 does not apply to HDFS-13891. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12957727/HDFS-14117-HDFS-13891.007.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26159/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761092#comment-16761092 ] Íñigo Goiri commented on HDFS-14117: Just a few minor comments: * {{RouterClientProtocol#478}} has too many parenthesis for the first part of the if. * Add a break line before {{RouterClientProtocol#createTrash()}} (before the javadoc). * The log message in {{RouterClientProtocol#509}}, it should be: {{LOG.debug("Creating Trash path: {}", location);}}. * We should also grab the output from {{invokeConcurrent()}} and check it while outputting success (debug) or error if something went wrong. * Break line in {{TestRouterRpc#492}}. * Break line in {{TestRouterRpc#1481}}. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760738#comment-16760738 ] venkata ramkumar commented on HDFS-14117: - Hi [~elgoiri] Uploaded the patch based on review comments.Please review it. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117-HDFS-13891.006.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760717#comment-16760717 ] Hadoop QA commented on HDFS-14117: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 31s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 5s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 49s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 13s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 37s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 52s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 49s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12957609/HDFS-14117-HDFS-13891.006.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4455b57523ab 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / b9d94c7 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26153/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26153/testReport/ | | Max. process+thread count | 1361 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759877#comment-16759877 ] venkata ramkumar commented on HDFS-14117: - Will upload the patch by fixing the failed tests cases, check style issues and find bug issues soon > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759872#comment-16759872 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 24s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 24s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 58s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 55s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 18s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf | | | Invocation of toString on options in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) At RouterClientProtocol.java:in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) At RouterClientProtocol.java:[line 478] | | | Call to String.equals(org.apache.hadoop.fs.Options$Rename) in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) At RouterClientProtocol.java:String, Options$Rename[]) At RouterClientProtocol.java:[line 478] | | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpc | | | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758532#comment-16758532 ] Íñigo Goiri commented on HDFS-14117: In addition to the failed unit test, checkstyles and line issues: * Avoid line change in {{RouterClientProtocol#21}}. * In {{RouterClientProtocol#477}} use {{!isEmpty()}}. Probably we should extract {{Arrays.asList(options)}} as it is used twice. * Avoid extra line in {{RouterClientProtocol#509}}. * You can have the declaration of {{RemoteMethod method}} in 497 as it does not change for each location. * We may want to add a debug message in {{createTrash()}}; similarly we may want to get the output of the Trash creation and log if there are errors. * In {{TestRouterRpcMultiDestination}}, the javadoc should be before the method and it should describe the end goal. * Even though {{TestRouterRpc}} is not really affected by this fix (as it is only one subcluster), I think we can take this chance to add a unit test here. Hopefully, we can leverage the function you already added. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758385#comment-16758385 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 16s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 19s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 53s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 15s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 53s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 78m 10s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12957258/HDFS-14117-HDFS-13891.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7c7d19f26779 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / d37590b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26111/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/26111/artifact/out/whitespace-eol.txt | | unit |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758272#comment-16758272 ] venkata ramkumar commented on HDFS-14117: - Hi [~elgoiri] Uploaded the patch based on review comments.Please review it. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117-HDFS-13891.004.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749504#comment-16749504 ] venkata ramkumar commented on HDFS-14117: - Thanks [~elgoiri] for the comment . Sure, I will make changes based on your comments. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749049#comment-16749049 ] Íñigo Goiri commented on HDFS-14117: Thanks [~ramkumar] for [^HDFS-14117-HDFS-13891.003.patch]. {quote} For this i don't think we need to test the Trash without multiple destinations . because with single destinations trash is working fine . i faced this problem only when its having multiple destinations. {quote} Not an issue but it is good to have coverage for all features. Given that you are already testing it for multiple, it should be trivial to move it to the base and extend if so in the rest. Minor comments: * {{RouterClientProtocol#102}} add an extra break line. * For the javadoc in {{RouterClientProtocol#movetoTrash()}}, make the parameters and throw description in the same line. * {{RouterClientProtocol#movetoTrash()}}, should probably be {{RouterClientProtocol#createTrash()}}. * Instead of going location by location to create the trash, I think you can use {{invokeConcurrent()}}. * {{TestRouterRpc#491}} add an extra break line. * {{TestRouterRpcMultiDestination#276}} add an extra break line. * Make the comment in {{TestRouterRpcMultiDestination#testTrash()}} a javadoc. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749029#comment-16749029 ] venkata ramkumar commented on HDFS-14117: - Thanks [~elgoiri] for the comment. bq. I think we should test the Trash without multiple destinations too. For this i don't think we need to test the Trash without multiple destinations . because with single destinations trash is working fine . i faced this problem only when its having multiple destinations. bq. Is there a way to test the Trash from a filesystem instead of relying directly on rename2? As of now i am not sure. I will try to check any other way for testing the trash from file system . If you have any idea regarding this please let know. If i missed any point in your previous comment i will try to cover in v4. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749014#comment-16749014 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} HDFS-14117 does not apply to HDFS-13891. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12955839/HDFS-14117-HDFS-13891.003.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26026/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch, > HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch, > HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748385#comment-16748385 ] venkata ramkumar commented on HDFS-14117: - Thanks Íñigo Goiri for reviewing the patch. I will make changes based on your comments and i will upload the patch soon. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748181#comment-16748181 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 1s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 24s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 37s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 53s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 32s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 42s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 82m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12955686/HDFS-14117-HDFS-13891.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 16880fa8fc62 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / 6ee477b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/26019/artifact/out/whitespace-eol.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26019/testReport/ | | Max. process+thread count | 977 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26019/console | | Powered by | Apache Yetus 0.8.0
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748140#comment-16748140 ] Íñigo Goiri commented on HDFS-14117: Thanks [~ramkumar] for [^HDFS-14117-HDFS-13891.001.patch]. Minor comments: * Avoid the reordering of the imports. * Fix the checstyle. * Fix the white space errors. * Rename the function to {{moveToTrash()}}. * Reformat the javadoc in {{moveToTrash()}} as the others and add the dot for the first parameter. * Rename new test to {{TestRouterRpc#testTrash()}}. * Make the capitalization of the comments in {{testTrash()}} consistent. * I think we should test the Trash without multiple destinations too. * Add the Test annotation to {{TestRouterRpcMultiDestination#testTrash()}}. * Add a javadoc to {{TestRouterRpcMultiDestination#testTrash()}} presenting the two cases instead of in the first comment. * Is there a way to test the Trash from a filesystem instead of relying directly on rename2? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117-HDFS-13891.001.patch, > HDFS-14117-HDFS-13891.002.patch, HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747970#comment-16747970 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-13891 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 8s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 45s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 49s{color} | {color:green} HDFS-13891 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} HDFS-13891 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 51s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 42s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 70m 38s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12955651/HDFS-14117-HDFS-13891.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b26daf66932f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-13891 / 6ee477b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26017/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/26017/artifact/out/whitespace-eol.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26017/testReport/ | | Max. process+thread count | 1373 (vs.
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747650#comment-16747650 ] venkata ramkumar commented on HDFS-14117: - Thanks Íñigo Goiri for the comment. Sure, I will do it. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746626#comment-16746626 ] Íñigo Goiri commented on HDFS-14117: I moved this to HDFS-13891. Can you rebase and fix the issues raised by Yetus? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746188#comment-16746188 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 1s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 13s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 10 new + 12 unchanged - 0 fixed = 22 total (was 12) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 13s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 57s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 30s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 53s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf | | | Possible null pointer dereference of options in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) Dereferenced at RouterClientProtocol.java:options in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) Dereferenced at RouterClientProtocol.java:[line 479] | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12955379/HDFS-14117.005.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1337a401ff30 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / dacc1a7 | | maven | version: Apache Maven 3.3.9 | | Default Java |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745883#comment-16745883 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 24s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 9 new + 12 unchanged - 0 fixed = 21 total (was 12) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 0s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 49s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 70m 30s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf | | | Possible null pointer dereference of options in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) Dereferenced at RouterClientProtocol.java:options in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) Dereferenced at RouterClientProtocol.java:[line 479] | | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12955345/HDFS-14117.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 09a599f09539 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745844#comment-16745844 ] venkata ramkumar commented on HDFS-14117: - Thanks [~elgoiri]for reviewing the patch.Uploaded the patch based on review comments.Please check once and let me know if any changes are required. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch, HDFS-14117.004.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740607#comment-16740607 ] Íñigo Goiri commented on HDFS-14117: {quote} And i am waiting for his comments to make another v4. {quote} Just move the new code block from {{RouterClientProtocol}} into a function with a full javadoc. Ideally, I would have a unit tes tin that function with a few full paths (e.g., /user/user1/.Trash) and bad cases. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740055#comment-16740055 ] venkata ramkumar commented on HDFS-14117: - Thanks [~ayushtkn] for the comment.Please check the message properly before commenting. Here [~elgoiri] has mentioned {quote}I would try to move some of this to a small function that we can unit test by itself with some simple example; in particular passing a trash path and checking the substring. {quote} And i am waiting for his comments to make another v4. And this one too {quote}Can we add this test to some other? Maybe the TestRouterRpcMultipleDestination? {quote} For this . Yes i missed because i want to have a discussion with [~elgoiri]. Since i already written test case and it will be difficult for me to again move this to TestRouterRpcMultipleDestination. I am waiting for the Jenkins result and also comments from [~elgoiri] regarding this. Putting up test classes for one-one test and setup up the whole environment doesn't seem good to me too.I Second this.We shouldn't volunteer making the tests so expensive. {quote}wait(4); {quote} What we can do here instead of wait. if you have any other solution please comment. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740025#comment-16740025 ] Ayush Saxena commented on HDFS-14117: - Thanx [~ramkumar] I guess you have created v3 as diff to v2 for TestRouterTrashpath.java. I guess you missed quite a lote of [~elgoiri] comments too. bq. I would try to move some of this to a small function that we can unit test by itself with some simple example; in particular passing a trash path and checking the substring. And this one too bq. Can we add this test to some other? Maybe the TestRouterRpcMultipleDestination? Putting up test classes for one-one test and setup up the whole environment doesn't seem good to me too.I Second this.We shouldn't volunteer making the tests so expensive. {code:java} wait(4); {code} Didn't get this too.Something like GenericTestUtils.waitFor(..) ??? {code:java} private static final Logger LOG = LoggerFactory 58.getLogger(TestRouterTrashpath.class); {code} In the test I see this up but didn't find it being used.Can you help.Might be due to patch isn't unified. I still see couple of Checkstyle still existing.Pls give a check to them too. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740010#comment-16740010 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} HDFS-14117 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12954522/HDFS-14117.003.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/25964/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ramkumar ch >Assignee: venkata ramkumar ch >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch, > HDFS-14117.003.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737829#comment-16737829 ] venkata ram kumar ch commented on HDFS-14117: - Thanks [~elgoiri] for reviewing the patch. I will make changes based on your comments and i will upload the patch soon. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ram kumar ch >Assignee: venkata ram kumar ch >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737405#comment-16737405 ] Íñigo Goiri commented on HDFS-14117: Thanks [~ramkumar] for the patch, it clarifies the JIRA a lot. Few comments: * Can we add a comment to the new code block in {{RouterClientProtocol}}; something like "Create the trash directory in each subcluster". * Can we add a debug log when we do this? * Does it make sense to do a check whether the parent was created? If not, can we log when we actually create the directories? * License issue. * Not sure what this findbug error means; can you take a look? * I would try to move some of this to a small function that we can unit test by itself with some simple example; in particular passing a trash path and checking the substring. * Can we add this test to some other? Maybe the {{TestRouterRpcMultipleDestination}}? * Can we separate a little the unit test (maybe just break lines) for the setup part and the actual test? * Rename the test to {{testTrash()}}? * Get rid of the sleep and actually wait for a condition. * For {{assertEquals(true)}} use {{assertTrue()}}. * Can we use the delete with trash option instead of rename2? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ram kumar ch >Assignee: venkata ram kumar ch >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch, HDFS-14117.002.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737246#comment-16737246 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 16s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 8 new + 12 unchanged - 0 fixed = 20 total (was 12) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 13s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 0s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 9s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 27s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf | | | Possible null pointer dereference of options in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) Dereferenced at RouterClientProtocol.java:options in org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.rename2(String, String, Options$Rename[]) Dereferenced at RouterClientProtocol.java:[line 491] | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12954149/HDFS-14117.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 71f67eb36dbc 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 695e93c | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | |
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736939#comment-16736939 ] Hadoop QA commented on HDFS-14117: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} HDFS-14117 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-14117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12954135/HDFS-14117.001.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/25928/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ram kumar ch >Assignee: venkata ram kumar ch >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736922#comment-16736922 ] venkata ram kumar ch commented on HDFS-14117: - Added the initial patch. Please check it once. > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ram kumar ch >Assignee: venkata ram kumar ch >Priority: Major > Labels: RBF > Attachments: HDFS-14117.001.patch > > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[ https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705034#comment-16705034 ] Íñigo Goiri commented on HDFS-14117: What we internally do is to mount the {{.Trash}} folder (e.g., {{/user/hadoop/.Trash}}). It's a little hacky to set it up this way but it should work. I don't get why the example reported in the description fails for /tmp but works for /test. Do you guys have a unit test to repro this? > RBF: We can only delete the files or dirs of one subcluster in a cluster with > multiple subclusters when trash is enabled > > > Key: HDFS-14117 > URL: https://issues.apache.org/jira/browse/HDFS-14117 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: venkata ram kumar ch >Assignee: Surendra Singh Lilhore >Priority: Major > Labels: RBF > > When we delete files or dirs in hdfs, it will move the deleted files or dirs > to trash by default. > But in the global path we can only mount one trash dir /user. So we mount > trash dir /user of the subcluster ns1 to the global path /user. Then we can > delete files or dirs of ns1, but when we delete the files or dirs of another > subcluser, such as hacluster, it will be failed. > h1. Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42| > |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: > -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05| > |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota: > -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20| > commands: > {noformat} > 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/. > 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd > 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/. > 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd > 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm > /tmp/mapred.cmd > 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination > parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found. > 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd > 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://router/test/hdfs.cmd' to trash at: > hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org