[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904774#comment-16904774
 ] 

Masatake Iwasaki commented on HDFS-14423:
-----------------------------------------

client deubg log shows expected URL in which "+" is not converted. 
{noformat}
$ bin/hdfs dfs -touchz 'webhdfs://localhost/a+b'
2019-08-12 09:10:04,929 TRACE web.WebHdfsFileSystem: 
url=http://localhost:9870/webhdfs/v1/a+b?op=GETFILESTATUS&user.name=iwasakims
2019-08-12 09:10:06,896 TRACE web.WebHdfsFileSystem: 
url=http://localhost:9870/webhdfs/v1/?op=GETFILESTATUS&user.name=iwasakims
2019-08-12 09:10:06,995 TRACE web.WebHdfsFileSystem: 
url=http://localhost:9870/webhdfs/v1/a+b?op=CREATE&user.name=iwasakims&blocksize=134217728&buffersize=4096&overwrite=true&permission=644&replication=1&unmaskedpermission=666
{noformat}

namenode log shows that GETFILESTATUS (for checking whether the file already 
exists) was applied to "a b" because URLDecoder#decode is called in 
NamenodeWebHdfsMethods#get.
{noformat}
2019-08-12 09:10:06,638 DEBUG 
org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger: ------------------- 
logged event for top service: allowed=true ugi=iwasakims (auth:SIMPLE)     
ip=/127.0.0.1   cmd=getfileinfo src=/a b        dst=null        per
m=null
2019-08-12 09:10:06,639 TRACE 
org.apache.hadoop.hdfs.web.resources.ExceptionHandler: GOT EXCEPITION
java.io.FileNotFoundException: File does not exist: /a b
        at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:1158)
        at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$4.run(NamenodeWebHdfsMethods.java:1075)
        at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$4.run(NamenodeWebHdfsMethods.java:1070)
        at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:78)
        at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2921)
{noformat}

Succeeding CREATE was given "a+b" as path since NamenodeWebHdfsMethods#put does 
not calls URLDecoder#decode. NameNode just redirect the request to chosen 
DataNode.
{noformat}
2019-08-12 09:10:07,016 TRACE 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods: 
HTTP PUT: op=CREATE, path=a+b, ugi=iwasakims (auth:SIMPLE), 
user.name=iwasakims, doas=null, accesstime=-1, blocksize=134217728, 
buffersize=4096, createflag=, flag=, modificationtime=-1, noredirect=false, 
overwrite=true, permission=644, renameoptions=, replication=1, 
unmaskedpermission=666
2019-08-12 09:10:07,051 TRACE 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods: 
redirectURI=http://localhost:9864/webhdfs/v1/a+b?op=CREATE&user.name=iwasakims&namenoderpcaddress=localhost:8020&blocksize=134217728&buffersize=4096&createflag=&createparent=true&overwrite=true&permission=644&replication=1&unmaskedpermission=666
{noformat}

DFSClient on redirected datanode created "a b" because 
WebHdfsHandler#channelRead0 always calls URLDecoder#decode on the path.
{noformat}
2019-08-12 09:10:07,725 DEBUG org.apache.hadoop.hdfs.DFSClient: /a b: masked={ 
masked: rw-r--r--, unmasked: rw-rw-rw- }
2019-08-12 09:10:07,860 DEBUG org.apache.hadoop.hdfs.DFSClient: 
computePacketChunkSize: src=/a b, chunkSize=516, chunksPerPacket=126, 
packetSize=65016
2019-08-12 09:10:07,867 DEBUG org.apache.hadoop.hdfs.client.impl.LeaseRenewer: 
Lease renewer daemon for [DFSClient_NONMAPREDUCE_683464448_30] with renew id 1 
started
2019-08-12 09:10:07,869 INFO datanode.webhdfs: 127.0.0.1 PUT 
/webhdfs/v1/a+b?op=CREATE&user.name=iwasakims&namenoderpcaddress=localhost:8020&blocksize=134217728&buffersize=4096&createflag=&createparent=true&overwrite=true&permission=644&replication=1&unmaskedpermission=666
 201
{noformat}

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -------------------------------------------------------------
>
>                 Key: HDFS-14423
>                 URL: https://issues.apache.org/jira/browse/HDFS-14423
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 3.2.0, 3.1.2
>         Environment: Ubuntu 16.04, but I believe this is irrelevant.
>            Reporter: Jing Wang
>            Assignee: Wei-Chiu Chuang
>            Priority: Major
>         Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup          0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup          0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to