[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907252#comment-16907252 ] Hadoop QA commented on HDFS-14423: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-2 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 39s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 1s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 59s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 41s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 59s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 17s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 6s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 26s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 10s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 33s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 7s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 59s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 35s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 18s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 42s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}166m 54s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | \\ \\ || Subsystem ||
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907140#comment-16907140 ] Masatake Iwasaki commented on HDFS-14423: - attached 006 addressing checksytle warnings. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-14423-branch-2.005.patch, > HDFS-14423-branch-2.006.patch, HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch, HDFS-14423.004.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907040#comment-16907040 ] Hadoop QA commented on HDFS-14423: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 45s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-2 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 57s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 32s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 52s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 44s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 49s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 15s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 52s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 1s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 5s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 5s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 45s{color} | {color:orange} root: The patch generated 1 new + 276 unchanged - 0 fixed = 277 total (was 276) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 52s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 38s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 13s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 42s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}173m 57s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906872#comment-16906872 ] Masatake Iwasaki commented on HDFS-14423: - Yeah. I'm working on backporting. I will attach branch-2 patch later. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch, HDFS-14423.004.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906775#comment-16906775 ] Akira Ajisaka commented on HDFS-14423: -- Would you backport this to branch-2? > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch, HDFS-14423.004.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906773#comment-16906773 ] Akira Ajisaka commented on HDFS-14423: -- Thank you, [~iwasakims]! > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch, HDFS-14423.004.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906737#comment-16906737 ] Hudson commented on HDFS-14423: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17108 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17108/]) HDFS-14423. Percent (%) and plus (+) characters no longer work in (iwasakims: rev da0006fe0473e353ee2d489156248a01aa982dfd) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch, HDFS-14423.004.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906708#comment-16906708 ] Masatake Iwasaki commented on HDFS-14423: - I'm committing this. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch, HDFS-14423.004.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906352#comment-16906352 ] Hadoop QA commented on HDFS-14423: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 40s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 17s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 40s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 42s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}202m 6s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14423 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12977481/HDFS-14423.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 21c36e2aaee5 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 0b507d2 | | maven | version:
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906141#comment-16906141 ] Masatake Iwasaki commented on HDFS-14423: - Thanks [~aajisaka]. I attached 004 addressing your comment. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch, HDFS-14423.004.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905989#comment-16905989 ] Akira Ajisaka commented on HDFS-14423: -- Thanks [~iwasakims] for digging into this issue. Minor comment: * You can use try-with-resources statement for MiniDFSCluster. I'm +1 if that is addressed. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905818#comment-16905818 ] Masatake Iwasaki commented on HDFS-14423: - {quote}That's worthy of revert but hopefully we won't wade through the debate of being compatible with the revert of an incompatible bug. {quote} The fix effectively reverts HDFS-13176. It is incompatible change for WebHdfsFileSystem(client) of 3.2.0, 3.1.0, 3.1.1 (and 2.10.0-SNAPSHOT) which uses encoded path in URL if ";" or "%" are contained. I think it is ok because webhdfs of those versions are just broken as reported here. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905815#comment-16905815 ] Masatake Iwasaki commented on HDFS-14423: - The result of manual testing: {noformat} [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 'webhdfs://localhost/%' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 'webhdfs://localhost/a+b' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 'webhdfs://localhost/a%b' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 'webhdfs://localhost/a;b' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -mkdir 'webhdfs://localhost/c+d' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -mkdir 'webhdfs://localhost/c%d' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -mkdir 'webhdfs://localhost/c;d' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hdfs dfs -ls / Found 7 items -rw-r--r-- 1 centos supergroup 0 2019-08-13 13:44 /% -rw-r--r-- 1 centos supergroup 0 2019-08-13 13:44 /a%b -rw-r--r-- 1 centos supergroup 0 2019-08-13 13:44 /a+b -rw-r--r-- 1 centos supergroup 0 2019-08-13 13:44 /a;b drwxr-xr-x - centos supergroup 0 2019-08-13 13:44 /c%d drwxr-xr-x - centos supergroup 0 2019-08-13 13:44 /c+d drwxr-xr-x - centos supergroup 0 2019-08-13 13:44 /c;d [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ echo foobar > foobar.txt [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -rm 'webhdfs://localhost/%' Deleted webhdfs://localhost/% [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -put foobar.txt 'webhdfs://localhost/%' [centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -cat 'webhdfs://localhost/%' foobar {noformat} > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905756#comment-16905756 ] Hadoop QA commented on HDFS-14423: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 11s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 1s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 19s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 17s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}199m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap | | | hadoop.hdfs.server.datanode.TestLargeBlockReport | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14423 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12977414/HDFS-14423.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 40e5a437c89c 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | |
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905670#comment-16905670 ] Masatake Iwasaki commented on HDFS-14423: - Attached 003 with fix for compilation issue. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, > HDFS-14423.003.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905501#comment-16905501 ] Wei-Chiu Chuang commented on HDFS-14423: Thanks for helping out. [~iwasakims] I assign this jira to you. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Masatake Iwasaki >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905284#comment-16905284 ] Hadoop QA commented on HDFS-14423: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 0s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 4s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 20s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 54s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m 32s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 32s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 16s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m 3s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m 57s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 20s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 17s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 59s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 98m 3s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 | | JIRA Issue | HDFS-14423 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12977358/HDFS-14423.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 6a6570ebe9ec 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / e4b538b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | mvninstall |
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905179#comment-16905179 ] Masatake Iwasaki commented on HDFS-14423: - I attached 002 attempting to refactor HDFS-13176. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905101#comment-16905101 ] Masatake Iwasaki commented on HDFS-14423: - NameNode is using Jersey for webhdfs while DataNode is using Netty. For NameNode (Jersey), setting [ResourceConfig.FEATURE_MATCH_MATRIX_PARAMS|https://jersey.github.io/apidocs/1.19.1/jersey/com/sun/jersey/api/core/ResourceConfig.html#FEATURE_MATCH_MATRIX_PARAMS] to "true" seems to avoid stripping after ";" in path element. For DataNode (Netty), ";" in path element is stripped by [QueryStringDecoder|https://netty.io/4.0/api/io/netty/handler/codec/http/QueryStringDecoder.html] for which I could not find tuning knob. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904933#comment-16904933 ] Masatake Iwasaki commented on HDFS-14423: - {quote} Those 2 referenced patches appear completely broken. {quote} I feel the same way now. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904926#comment-16904926 ] Masatake Iwasaki commented on HDFS-14423: - {quote}client deubg log shows expected URL in which "+" is not converted. $ bin/hdfs dfs -touchz 'webhdfs://localhost/a+b' 2019-08-12 09:10:04,929 TRACE web.WebHdfsFileSystem: url=[http://localhost:9870/webhdfs/v1/a+b?op=GETFILESTATUS=iwasakims] {quote} I think even this was not so expected result now. Since SPECIAL_FILENAME_CHARACTERS_REGEX matches to "a+b", is is applied to URLEncoder#encode. {noformat} public static final String SPECIAL_FILENAME_CHARACTERS_REGEX = ".*[;+%].*"; {noformat} The "+" in the URL is not encoded because "a+b" is converted into "a b" by URLDecoder#decode first then encoded back into "a+b" by URLEncoder#encode. The code intended for already url-encoded path elements seems to be working in unintended way. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904912#comment-16904912 ] Masatake Iwasaki commented on HDFS-14423: - For path containing "%" like "webhdfs://localhost/%", the percent converted into "%25" by URLEncoder#encode then into "%2525" by constructor of URI. {noformat} 2019-08-12 15:36:53,962 TRACE web.WebHdfsFileSystem: url=http://localhost:9870/webhdfs/v1/%2525?op=GETFILESTATUS=iwasakims {noformat} This is incompatible with 2.9 in which "%" is encoded once by constructor of URI. {noformat} $ bin/hdfs dfs -touchz 'webhdfs://localhost/%' 19/08/12 16:07:32 DEBUG web.WebHdfsFileSystem: Not enabling OAuth2 in WebHDFS 19/08/12 16:07:32 TRACE web.WebHdfsFileSystem: url=http://localhost:50070/webhdfs/v1/%25?op=GETFILESTATUS=iwasakims {noformat} > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904774#comment-16904774 ] Masatake Iwasaki commented on HDFS-14423: - client deubg log shows expected URL in which "+" is not converted. {noformat} $ bin/hdfs dfs -touchz 'webhdfs://localhost/a+b' 2019-08-12 09:10:04,929 TRACE web.WebHdfsFileSystem: url=http://localhost:9870/webhdfs/v1/a+b?op=GETFILESTATUS=iwasakims 2019-08-12 09:10:06,896 TRACE web.WebHdfsFileSystem: url=http://localhost:9870/webhdfs/v1/?op=GETFILESTATUS=iwasakims 2019-08-12 09:10:06,995 TRACE web.WebHdfsFileSystem: url=http://localhost:9870/webhdfs/v1/a+b?op=CREATE=iwasakims=134217728=4096=true=644=1=666 {noformat} namenode log shows that GETFILESTATUS (for checking whether the file already exists) was applied to "a b" because URLDecoder#decode is called in NamenodeWebHdfsMethods#get. {noformat} 2019-08-12 09:10:06,638 DEBUG org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger: --- logged event for top service: allowed=true ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/a bdst=nullper m=null 2019-08-12 09:10:06,639 TRACE org.apache.hadoop.hdfs.web.resources.ExceptionHandler: GOT EXCEPITION java.io.FileNotFoundException: File does not exist: /a b at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:1158) at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$4.run(NamenodeWebHdfsMethods.java:1075) at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$4.run(NamenodeWebHdfsMethods.java:1070) at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:78) at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2921) {noformat} Succeeding CREATE was given "a+b" as path since NamenodeWebHdfsMethods#put does not calls URLDecoder#decode. NameNode just redirect the request to chosen DataNode. {noformat} 2019-08-12 09:10:07,016 TRACE org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods: HTTP PUT: op=CREATE, path=a+b, ugi=iwasakims (auth:SIMPLE), user.name=iwasakims, doas=null, accesstime=-1, blocksize=134217728, buffersize=4096, createflag=, flag=, modificationtime=-1, noredirect=false, overwrite=true, permission=644, renameoptions=, replication=1, unmaskedpermission=666 2019-08-12 09:10:07,051 TRACE org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods: redirectURI=http://localhost:9864/webhdfs/v1/a+b?op=CREATE=iwasakims=localhost:8020=134217728=4096==true=true=644=1=666 {noformat} DFSClient on redirected datanode created "a b" because WebHdfsHandler#channelRead0 always calls URLDecoder#decode on the path. {noformat} 2019-08-12 09:10:07,725 DEBUG org.apache.hadoop.hdfs.DFSClient: /a b: masked={ masked: rw-r--r--, unmasked: rw-rw-rw- } 2019-08-12 09:10:07,860 DEBUG org.apache.hadoop.hdfs.DFSClient: computePacketChunkSize: src=/a b, chunkSize=516, chunksPerPacket=126, packetSize=65016 2019-08-12 09:10:07,867 DEBUG org.apache.hadoop.hdfs.client.impl.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_683464448_30] with renew id 1 started 2019-08-12 09:10:07,869 INFO datanode.webhdfs: 127.0.0.1 PUT /webhdfs/v1/a+b?op=CREATE=iwasakims=localhost:8020=134217728=4096==true=true=644=1=666 201 {noformat} > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904735#comment-16904735 ] Masatake Iwasaki commented on HDFS-14423: - https://tools.ietf.org/html/rfc3986#section-3.3 {quote} URI producing applications often use the reserved characters allowed in a segment to delimit scheme-specific or dereference-handler-specific subcomponents. For example, the semicolon (";") and equals ("=") reserved characters are often used to delimit parameters and parameter values applicable to that segment. The comma (",") reserved character is often used for similar purposes. For example, one URI producer might use a segment such as "name;v=1.1" to indicate a reference to version 1.1 of "name", whereas another might use a segment such as "name,1.1" to indicate the same. Parameter types may be defined by scheme-specific semantics, but in most cases the syntax of a parameter is specific to the implementation of the URI's dereferencing algorithm. {quote} {noformat} reserved= gen-delims / sub-delims gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@" sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" ... path = path-abempty; begins with "/" or is empty / path-absolute ; begins with "/" but not "//" / path-noscheme ; begins with a non-colon segment / path-rootless ; begins with a segment / path-empty ; zero characters path-abempty = *( "/" segment ) path-absolute = "/" [ segment-nz *( "/" segment ) ] path-noscheme = segment-nz-nc *( "/" segment ) path-rootless = segment-nz *( "/" segment ) path-empty= 0 segment = *pchar segment-nz= 1*pchar segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" ) ; non-zero-length segment without any colon ":" pchar = unreserved / pct-encoded / sub-delims / ":" / "@" {noformat} > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904731#comment-16904731 ] Masatake Iwasaki commented on HDFS-14423: - The reason why WebHdfs file path gets truncated when having semicolon (;) inside is that Jersey treats the part after ";" as parameters. I think using URLEncoder/URLDecoder is not the right way for HDFS-13176. I'm going to try refatctoring here. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904728#comment-16904728 ] Masatake Iwasaki commented on HDFS-14423: - java.net.URLEncoder converts "\+" into " " (since java.net.URLEncoder converts " " into "\+"). The code path of {{open}} seems to convert "a+b" into "a b" while {{mkdir}} does not. > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904531#comment-16904531 ] Masatake Iwasaki commented on HDFS-14423: - Thanks [~jojochuang]. While the patch looks good, the issued reported as "plus (+ ) characters get turned into spaces" still seems to be exists. I'm looking into the cause. {noformat} [iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz webhdfs://localhost/% [iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz webhdfs://localhost/a+b [iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -mkdir webhdfs://localhost/c+d [iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls webhdfs://localhost/ [iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hadoop fs -put 'e+f.txt' 'webhdfs://localhost/' put: rename `webhdfs://localhost/e+f.txt._COPYING_' to `webhdfs://localhost/e+f.txt': Input/output error [iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls / Found 6 items -rw-r--r-- 1 iwasakims supergroup 0 2019-08-09 23:35 /% -rw-r--r-- 1 iwasakims supergroup 0 2019-08-09 23:50 /a b drwxr-xr-x - iwasakims supergroup 0 2019-08-09 23:37 /c+d -rw-r--r-- 1 iwasakims supergroup 5 2019-08-09 23:52 /e f.txt._COPYING_ drwxrwxrwx - iwasakims supergroup 0 2019-06-14 13:09 /tmp drwxr-xr-x - iwasakims supergroup 0 2019-06-14 13:10 /user {noformat} > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904263#comment-16904263 ] Hadoop QA commented on HDFS-14423: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 57s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 11s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}138m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport | | | hadoop.hdfs.TestRollingUpgrade | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14423 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12977193/HDFS-14423.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b228ccd437e0 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 98dd7c4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/27462/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/27462/testReport/ | | Max. process+thread count | 4084 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/27462/console | | Powered by | Apache Yetus 0.8.0
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904097#comment-16904097 ] Wei-Chiu Chuang commented on HDFS-14423: I updated TestWebHdfsUrl which was added in HDFS-14466 to verify this was fixed by HDFS-14323. Attached a patch to update the test to verify special character + is handled correctly as well. [~iwasakims] [~aajisaka] mind to review this trivial patch? > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-14423.001.patch > > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899384#comment-16899384 ] Wei-Chiu Chuang commented on HDFS-14423: Does HDFS-14323 fix it? > Percent (%) and plus (+) characters no longer work in WebHDFS > - > > Key: HDFS-14423 > URL: https://issues.apache.org/jira/browse/HDFS-14423 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 > Environment: Ubuntu 16.04, but I believe this is irrelevant. >Reporter: Jing Wang >Priority: Major > > The following commands with percent (%) no longer work starting with version > 3.1: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/% > $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/% > cat: URLDecoder: Incomplete trailing escape (%) pattern > {code} > Also, plus (+ ) characters get turned into spaces when doing DN operations: > {code:java} > $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b > $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d > $ hadoop/bin/hdfs dfs -ls / > Found 4 items > -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b > drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d > {code} > I can confirm that these commands work correctly on 2.9 and 3.0. Also, the > usual hdfs:// client works as expected. > I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the > right fix is. Note that Hive uses % to escape special characters in partition > values, so banning % might not be a good option. For example, Hive will > create a paths like {{table_name/partition_key=%2F}} when > {{partition_key='/'}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org