[jira] [Commented] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-10 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948572#comment-16948572
 ] 

Masatake Iwasaki commented on HDFS-14900:
-

I need to fix Dockerfile too. attached 003.

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14900.001.patch, HDFS-14900.002.patch, 
> HDFS-14900.003.patch
>
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-10 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14900:

Attachment: HDFS-14900.003.patch

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14900.001.patch, HDFS-14900.002.patch, 
> HDFS-14900.003.patch
>
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-10 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948564#comment-16948564
 ] 

Masatake Iwasaki commented on HDFS-14900:
-

attached 002 fixing only BUILDING.txt.

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14900.001.patch, HDFS-14900.002.patch
>
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-10 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14900:

Attachment: HDFS-14900.002.patch

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14900.001.patch, HDFS-14900.002.patch
>
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-09 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14900:

Status: Patch Available  (was: Open)

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14900.001.patch
>
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-09 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947424#comment-16947424
 ] 

Masatake Iwasaki commented on HDFS-14900:
-

I attached 001 fixing BUILDING.txt and CMakeList.txt. It makes build of 
libhdfspp skipped if libprotobuf is not found since only libhdfspp requires 
libprotobuf.

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14900.001.patch
>
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-09 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14900:

Attachment: HDFS-14900.001.patch

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14900.001.patch
>
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14900) Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf

2019-10-09 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14900:

Summary: Fix requirements of BUILDING.txt for libhdfspp which depends on 
libprotobuf  (was: Fix build failure of hadoop-hdfs-native-client)

> Fix requirements of BUILDING.txt for libhdfspp which depends on libprotobuf
> ---
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14900) Fix build failure of hadoop-hdfs-native-client

2019-10-09 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947391#comment-16947391
 ] 

Masatake Iwasaki commented on HDFS-14900:
-

I could not find a portable way to download libprotobuf and libprotoc and 
headers on build time automatically.

I thought copying files under 
https://github.com/protocolbuffers/protobuf/tree/v3.7.1/src to 
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/
 might be an option but [FindProtobuf module of 
cmake|https://cmake.org/cmake/help/v3.9/module/FindProtobuf.html] seems not to 
work with it.

I'm considering to add description about Protocol Buffers back to BUILDING.txt 
as an optional requirement for {{-Pnative}} build.

> Fix build failure of hadoop-hdfs-native-client
> --
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14900) Fix build failure of hadoop-hdfs-native-client

2019-10-09 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14900:

Description: HADOOP-16558 removed protocol buffers from build requirements 
but libhdfspp requires libprotobuf and libprotoc. {{-Pnative}} build fails if 
protocol buffers is not installed.  (was: Build with native profile failed due 
to lack of protobuf resources. I did not noticed this because build succeeds if 
protobuf-2.5.0 is installed. protobuf-3.7.1 is the correct version after 
HADOOP-16558.)

> Fix build failure of hadoop-hdfs-native-client
> --
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> HADOOP-16558 removed protocol buffers from build requirements but libhdfspp 
> requires libprotobuf and libprotoc. {{-Pnative}} build fails if protocol 
> buffers is not installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14900) Fix build failure of hadoop-hdfs-native-client

2019-10-07 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946505#comment-16946505
 ] 

Masatake Iwasaki commented on HDFS-14900:
-

{quote}
If I catch it correct now, you are telling, The build fails when no protobuf is 
installed explicitly?
{quote}

Yes. Sorry for unclear problem statement.

{quote}
Anyway if so, do you have fix for it?
{quote}

I'm looking for the way without manual protobuf installation.


> Fix build failure of hadoop-hdfs-native-client
> --
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> Build with native profile failed due to lack of protobuf resources. I did not 
> noticed this because build scceeds if protobuf-2.5.0 is installed. 
> protobuf-3.7.1 is the correct version after HADOOP-16558.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14900) Fix build failure of hadoop-hdfs-native-client

2019-10-07 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946484#comment-16946484
 ] 

Masatake Iwasaki commented on HDFS-14900:
-

> Tried on my system, seems working..

[~ayushtkn] If you have installed protobuf on your system, native build works. 
I think this jira is a follow-up of HADOOP-16620 which removed protobuf from 
build env.


> Fix build failure of hadoop-hdfs-native-client
> --
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> Build with native profile failed due to lack of protobuf resources. I did not 
> noticed this because build scceeds if protobuf-2.5.0 is installed. 
> protobuf-3.7.1 is the correct version after HADOOP-16558.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14900) Fix build failure of hadoop-hdfs-native-client

2019-10-07 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946311#comment-16946311
 ] 

Masatake Iwasaki commented on HDFS-14900:
-

from CMakeError.log:
{noformat}
/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=pthread_create   -o 
CMakeFiles/cmTC_cace7.dir/CheckFunctionExists.c.o   -c 
/usr/local/share/cmake-3.3/Modules/CheckFunctionExists.c
Linking C executable cmTC_cace7
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_cace7.dir/link.txt 
--verbose=1
/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=pthread_create
CMakeFiles/cmTC_cace7.dir/CheckFunctionExists.c.o  -o cmTC_cace7 -rdynamic 
-lpthreads 
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
{noformat}
{noformat}
Building CXX object CMakeFiles/cmTC_6c0d6.dir/src.cxx.o
/usr/bin/c++-g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE 
-DPROTOC_IS_COMPATIBLE   -std=c++11 -o CMakeFiles/cmTC_6c0d6.dir/src.cxx.o -c 
/home/iwasakims/srcs/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/CMakeFiles/CMakeTmp/src.cxx
/home/iwasakims/srcs/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/CMakeFiles/CMakeTmp/src.cxx:1:40:
 fatal error: google/protobuf/io/printer.h: No such file or directory
 #include 
{noformat}


> Fix build failure of hadoop-hdfs-native-client
> --
>
> Key: HDFS-14900
> URL: https://issues.apache.org/jira/browse/HDFS-14900
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> Build with native profile failed due to lack of protobuf resources. I did not 
> noticed this because build scceeds if protobuf-2.5.0 is installed. 
> protobuf-3.7.1 is the correct version after HADOOP-16558.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Created] (HDFS-14900) Fix build failure of hadoop-hdfs-native-client

2019-10-07 Thread Masatake Iwasaki (Jira)
Masatake Iwasaki created HDFS-14900:
---

 Summary: Fix build failure of hadoop-hdfs-native-client
 Key: HDFS-14900
 URL: https://issues.apache.org/jira/browse/HDFS-14900
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


Build with native profile failed due to lack of protobuf resources. I did not 
noticed this because build scceeds if protobuf-2.5.0 is installed. 
protobuf-3.7.1 is the correct version after HADOOP-16558.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-13699) Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP

2019-09-18 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932702#comment-16932702
 ] 

Masatake Iwasaki commented on HDFS-13699:
-

[~vagarychen] I'm updating fix versions since this seemed to be committed to 
branch-3.1 and branch-3.2.

> Add DFSClient sending handshake token to DataNode, and allow DataNode 
> overwrite downstream QOP
> --
>
> Key: HDFS-13699
> URL: https://issues.apache.org/jira/browse/HDFS-13699
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-13699.001.patch, HDFS-13699.002.patch, 
> HDFS-13699.003.patch, HDFS-13699.004.patch, HDFS-13699.005.patch, 
> HDFS-13699.006.patch, HDFS-13699.007.patch, HDFS-13699.008.patch, 
> HDFS-13699.009.patch, HDFS-13699.010.patch, HDFS-13699.WIP.001.patch
>
>
> Given the other Jiras under HDFS-13541, this Jira is to allow DFSClient to 
> redirect the encrypt secret to DataNode. The encrypted message is the QOP 
> that client and NameNode have used. DataNode decrypts the message and enforce 
> the QOP for the client connection. Also, this Jira will also include 
> overwriting downstream QOP, as mentioned in the HDFS-13541 design doc. 
> Namely, this is to allow inter-DN QOP that is different from client-DN QOP.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-13699) Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP

2019-09-18 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-13699:

Fix Version/s: 3.1.3
   3.2.1

> Add DFSClient sending handshake token to DataNode, and allow DataNode 
> overwrite downstream QOP
> --
>
> Key: HDFS-13699
> URL: https://issues.apache.org/jira/browse/HDFS-13699
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-13699.001.patch, HDFS-13699.002.patch, 
> HDFS-13699.003.patch, HDFS-13699.004.patch, HDFS-13699.005.patch, 
> HDFS-13699.006.patch, HDFS-13699.007.patch, HDFS-13699.008.patch, 
> HDFS-13699.009.patch, HDFS-13699.010.patch, HDFS-13699.WIP.001.patch
>
>
> Given the other Jiras under HDFS-13541, this Jira is to allow DFSClient to 
> redirect the encrypt secret to DataNode. The encrypted message is the QOP 
> that client and NameNode have used. DataNode decrypts the message and enforce 
> the QOP for the client connection. Also, this Jira will also include 
> overwriting downstream QOP, as mentioned in the HDFS-13541 design doc. 
> Namely, this is to allow inter-DN QOP that is different from client-DN QOP.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14759) HDFS cat logs an info message

2019-09-18 Thread Masatake Iwasaki (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932701#comment-16932701
 ] 

Masatake Iwasaki commented on HDFS-14759:
-

cherry-picked to branch-3.2 and branch-3.1.

> HDFS cat logs an info message
> -
>
> Key: HDFS-14759
> URL: https://issues.apache.org/jira/browse/HDFS-14759
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14759.001.patch
>
>
> HDFS-13699 changed a debug log line into an info log line and this line is 
> printed during {{hadoop fs -cat}} operations. This make it very difficult to 
> figure out where the log line ends and where the catted file begins, 
> especially when the output is sent to a tool for parsing. 
> {noformat}
> [ebadger@foobar bin]$ hadoop fs -cat /foo 2>/dev/null
> 2019-08-20 22:09:45,907 INFO  [main] sasl.SaslDataTransferClient 
> (SaslDataTransferClient.java:checkTrustAndSend(230)) - SASL encryption trust 
> check: localHostTrusted = false, remoteHostTrusted = false
> bar
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14759) HDFS cat logs an info message

2019-09-18 Thread Masatake Iwasaki (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14759:

Fix Version/s: 3.2.2
   3.1.4

> HDFS cat logs an info message
> -
>
> Key: HDFS-14759
> URL: https://issues.apache.org/jira/browse/HDFS-14759
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14759.001.patch
>
>
> HDFS-13699 changed a debug log line into an info log line and this line is 
> printed during {{hadoop fs -cat}} operations. This make it very difficult to 
> figure out where the log line ends and where the catted file begins, 
> especially when the output is sent to a tool for parsing. 
> {noformat}
> [ebadger@foobar bin]$ hadoop fs -cat /foo 2>/dev/null
> 2019-08-20 22:09:45,907 INFO  [main] sasl.SaslDataTransferClient 
> (SaslDataTransferClient.java:checkTrustAndSend(230)) - SASL encryption trust 
> check: localHostTrusted = false, remoteHostTrusted = false
> bar
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-14 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14707:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.10.0
   Status: Resolved  (was: Patch Available)

I committed this. Thanks for reviewing, [~aajisaka].

>  Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2
> 
>
> Key: HDFS-14707
> URL: https://issues.apache.org/jira/browse/HDFS-14707
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 2.10.0
>
> Attachments: HDFS-14707-branch-2.001.patch, 
> HDFS-14707-branch-2.002.patch
>
>
> Currently HTTPFS does not load hadoop native library since java.library.path 
> is not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-14 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14423:

   Resolution: Fixed
Fix Version/s: 2.10.0
   Status: Resolved  (was: Patch Available)

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14423-branch-2.005.patch, 
> HDFS-14423-branch-2.006.patch, HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-14 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14423:

Attachment: HDFS-14423-branch-2.006.patch

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14423-branch-2.005.patch, 
> HDFS-14423-branch-2.006.patch, HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-14 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907140#comment-16907140
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

attached 006 addressing checksytle warnings.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14423-branch-2.005.patch, 
> HDFS-14423-branch-2.006.patch, HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-14 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14423:

Attachment: HDFS-14423-branch-2.005.patch

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14423-branch-2.005.patch, HDFS-14423.001.patch, 
> HDFS-14423.002.patch, HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-13 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906872#comment-16906872
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

Yeah. I'm working on backporting. I will attach branch-2 patch later.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-13 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906708#comment-16906708
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

I'm committing this.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-13 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906141#comment-16906141
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

Thanks [~aajisaka]. I attached 004 addressing your comment.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-13 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14423:

Attachment: HDFS-14423.004.patch

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch, HDFS-14423.004.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-13 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906121#comment-16906121
 ] 

Masatake Iwasaki commented on HDFS-14707:
-

I updated the patch based on your comment. Thanks, [~aajisaka].

>  Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2
> 
>
> Key: HDFS-14707
> URL: https://issues.apache.org/jira/browse/HDFS-14707
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14707-branch-2.001.patch, 
> HDFS-14707-branch-2.002.patch
>
>
> Currently HTTPFS does not load hadoop native library since java.library.path 
> is not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-13 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14707:

Attachment: HDFS-14707-branch-2.002.patch

>  Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2
> 
>
> Key: HDFS-14707
> URL: https://issues.apache.org/jira/browse/HDFS-14707
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14707-branch-2.001.patch, 
> HDFS-14707-branch-2.002.patch
>
>
> Currently HTTPFS does not load hadoop native library since java.library.path 
> is not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905818#comment-16905818
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

{quote}That's worthy of revert but hopefully we won't wade through the debate 
of being compatible with the revert of an incompatible bug.
{quote}
The fix effectively reverts HDFS-13176. It is incompatible change for 
WebHdfsFileSystem(client) of 3.2.0, 3.1.0, 3.1.1 (and 2.10.0-SNAPSHOT) which 
uses encoded path in URL if ";" or "%" are contained. I think it is ok because 
webhdfs of those versions are just broken as reported here.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905815#comment-16905815
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

The result of manual testing:
{noformat}
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 
'webhdfs://localhost/%'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 
'webhdfs://localhost/a+b'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 
'webhdfs://localhost/a%b'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -touchz 
'webhdfs://localhost/a;b'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -mkdir 
'webhdfs://localhost/c+d'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -mkdir 
'webhdfs://localhost/c%d'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -mkdir 
'webhdfs://localhost/c;d'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hdfs dfs -ls /
Found 7 items
-rw-r--r--   1 centos supergroup  0 2019-08-13 13:44 /%
-rw-r--r--   1 centos supergroup  0 2019-08-13 13:44 /a%b
-rw-r--r--   1 centos supergroup  0 2019-08-13 13:44 /a+b
-rw-r--r--   1 centos supergroup  0 2019-08-13 13:44 /a;b
drwxr-xr-x   - centos supergroup  0 2019-08-13 13:44 /c%d
drwxr-xr-x   - centos supergroup  0 2019-08-13 13:44 /c+d
drwxr-xr-x   - centos supergroup  0 2019-08-13 13:44 /c;d

[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ echo foobar > foobar.txt
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -rm 
'webhdfs://localhost/%'
Deleted webhdfs://localhost/%
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -put 
foobar.txt 'webhdfs://localhost/%'
[centos@centos7 hadoop-3.3.0-SNAPSHOT-HDFS-14423]$ bin/hadoop fs -cat 
'webhdfs://localhost/%'
foobar
{noformat}

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904728#comment-16904728
 ] 

Masatake Iwasaki edited comment on HDFS-14423 at 8/13/19 4:11 AM:
--

java.net.URLDecoder converts "\+" into " " (since java.net.URLEncoder converts 
" " into "\+"). The code path of {{create}} seems to convert "a+b" into "a b" 
while {{mkdir}} does not.


was (Author: iwasakims):
java.net.URLDecoder converts "+" into " " (since java.net.URLEncoder converts " 
" into "+"). The code path of {{create}} seems to convert "a+b" into "a b" 
while {{mkdir}} does not.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904728#comment-16904728
 ] 

Masatake Iwasaki edited comment on HDFS-14423 at 8/13/19 4:10 AM:
--

java.net.URLDecoder converts "+" into " " (since java.net.URLEncoder converts " 
" into "+"). The code path of {{create}} seems to convert "a+b" into "a b" 
while {{mkdir}} does not.


was (Author: iwasakims):
java.net.URLEncoder converts "\+" into " " (since java.net.URLEncoder converts 
" " into "\+"). The code path of {{create}} seems to convert "a+b" into "a b" 
while {{mkdir}} does not.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905670#comment-16905670
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

Attached 003 with fix for compilation issue.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14423:

Attachment: HDFS-14423.003.patch

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch, 
> HDFS-14423.003.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905179#comment-16905179
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

I attached 002 attempting to refactor HDFS-13176.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14423:

Attachment: HDFS-14423.002.patch

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch, HDFS-14423.002.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905101#comment-16905101
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

NameNode is using Jersey for webhdfs while DataNode is using Netty.

For NameNode (Jersey), setting 
[ResourceConfig.FEATURE_MATCH_MATRIX_PARAMS|https://jersey.github.io/apidocs/1.19.1/jersey/com/sun/jersey/api/core/ResourceConfig.html#FEATURE_MATCH_MATRIX_PARAMS]
 to "true" seems to avoid stripping after ";" in path element.

For DataNode (Netty), ";" in path element is stripped by 
[QueryStringDecoder|https://netty.io/4.0/api/io/netty/handler/codec/http/QueryStringDecoder.html]
 for which I could not find tuning knob.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904933#comment-16904933
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

{quote}
Those 2 referenced patches appear completely broken.
{quote}

I feel the same way now.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904926#comment-16904926
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

{quote}client deubg log shows expected URL in which "+" is not converted.

$ bin/hdfs dfs -touchz 'webhdfs://localhost/a+b'
 2019-08-12 09:10:04,929 TRACE web.WebHdfsFileSystem: 
url=[http://localhost:9870/webhdfs/v1/a+b?op=GETFILESTATUS=iwasakims]
{quote}
I think even this was not so expected result now. Since 
SPECIAL_FILENAME_CHARACTERS_REGEX matches to "a+b", is is applied to 
URLEncoder#encode.
{noformat}
  public static final String SPECIAL_FILENAME_CHARACTERS_REGEX = ".*[;+%].*";
{noformat}
The "+" in the URL is not encoded because "a+b" is converted into "a b" by 
URLDecoder#decode first then encoded back into "a+b" by URLEncoder#encode. The 
code intended for already url-encoded path elements seems to be working in 
unintended way.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-12 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904912#comment-16904912
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

For path containing "%" like "webhdfs://localhost/%", the percent converted 
into "%25" by URLEncoder#encode then into "%2525" by constructor of URI.
{noformat}
2019-08-12 15:36:53,962 TRACE web.WebHdfsFileSystem: 
url=http://localhost:9870/webhdfs/v1/%2525?op=GETFILESTATUS=iwasakims
{noformat}
This is incompatible with 2.9 in which "%" is encoded once by constructor of 
URI.
{noformat}
$ bin/hdfs dfs -touchz 'webhdfs://localhost/%'
19/08/12 16:07:32 DEBUG web.WebHdfsFileSystem: Not enabling OAuth2 in WebHDFS
19/08/12 16:07:32 TRACE web.WebHdfsFileSystem: 
url=http://localhost:50070/webhdfs/v1/%25?op=GETFILESTATUS=iwasakims
{noformat}

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904728#comment-16904728
 ] 

Masatake Iwasaki edited comment on HDFS-14423 at 8/12/19 1:06 AM:
--

java.net.URLEncoder converts "\+" into " " (since java.net.URLEncoder converts 
" " into "\+"). The code path of {{create}} seems to convert "a+b" into "a b" 
while {{mkdir}} does not.


was (Author: iwasakims):
java.net.URLEncoder converts "\+" into " " (since java.net.URLEncoder converts 
" " into "\+"). The code path of {{open}} seems to convert "a+b" into "a b" 
while {{mkdir}} does not.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904774#comment-16904774
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

client deubg log shows expected URL in which "+" is not converted. 
{noformat}
$ bin/hdfs dfs -touchz 'webhdfs://localhost/a+b'
2019-08-12 09:10:04,929 TRACE web.WebHdfsFileSystem: 
url=http://localhost:9870/webhdfs/v1/a+b?op=GETFILESTATUS=iwasakims
2019-08-12 09:10:06,896 TRACE web.WebHdfsFileSystem: 
url=http://localhost:9870/webhdfs/v1/?op=GETFILESTATUS=iwasakims
2019-08-12 09:10:06,995 TRACE web.WebHdfsFileSystem: 
url=http://localhost:9870/webhdfs/v1/a+b?op=CREATE=iwasakims=134217728=4096=true=644=1=666
{noformat}

namenode log shows that GETFILESTATUS (for checking whether the file already 
exists) was applied to "a b" because URLDecoder#decode is called in 
NamenodeWebHdfsMethods#get.
{noformat}
2019-08-12 09:10:06,638 DEBUG 
org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger: --- 
logged event for top service: allowed=true ugi=iwasakims (auth:SIMPLE) 
ip=/127.0.0.1   cmd=getfileinfo src=/a bdst=nullper
m=null
2019-08-12 09:10:06,639 TRACE 
org.apache.hadoop.hdfs.web.resources.ExceptionHandler: GOT EXCEPITION
java.io.FileNotFoundException: File does not exist: /a b
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:1158)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$4.run(NamenodeWebHdfsMethods.java:1075)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$4.run(NamenodeWebHdfsMethods.java:1070)
at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:78)
at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2921)
{noformat}

Succeeding CREATE was given "a+b" as path since NamenodeWebHdfsMethods#put does 
not calls URLDecoder#decode. NameNode just redirect the request to chosen 
DataNode.
{noformat}
2019-08-12 09:10:07,016 TRACE 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods: 
HTTP PUT: op=CREATE, path=a+b, ugi=iwasakims (auth:SIMPLE), 
user.name=iwasakims, doas=null, accesstime=-1, blocksize=134217728, 
buffersize=4096, createflag=, flag=, modificationtime=-1, noredirect=false, 
overwrite=true, permission=644, renameoptions=, replication=1, 
unmaskedpermission=666
2019-08-12 09:10:07,051 TRACE 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods: 
redirectURI=http://localhost:9864/webhdfs/v1/a+b?op=CREATE=iwasakims=localhost:8020=134217728=4096==true=true=644=1=666
{noformat}

DFSClient on redirected datanode created "a b" because 
WebHdfsHandler#channelRead0 always calls URLDecoder#decode on the path.
{noformat}
2019-08-12 09:10:07,725 DEBUG org.apache.hadoop.hdfs.DFSClient: /a b: masked={ 
masked: rw-r--r--, unmasked: rw-rw-rw- }
2019-08-12 09:10:07,860 DEBUG org.apache.hadoop.hdfs.DFSClient: 
computePacketChunkSize: src=/a b, chunkSize=516, chunksPerPacket=126, 
packetSize=65016
2019-08-12 09:10:07,867 DEBUG org.apache.hadoop.hdfs.client.impl.LeaseRenewer: 
Lease renewer daemon for [DFSClient_NONMAPREDUCE_683464448_30] with renew id 1 
started
2019-08-12 09:10:07,869 INFO datanode.webhdfs: 127.0.0.1 PUT 
/webhdfs/v1/a+b?op=CREATE=iwasakims=localhost:8020=134217728=4096==true=true=644=1=666
 201
{noformat}

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 

[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904735#comment-16904735
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

https://tools.ietf.org/html/rfc3986#section-3.3
{quote}
URI producing applications often use the reserved characters allowed in a 
segment to delimit scheme-specific or dereference-handler-specific 
subcomponents.  For example, the semicolon (";") and equals ("=") reserved 
characters are often used to delimit parameters and parameter values applicable 
to that segment.  The comma (",") reserved character is often used for similar 
purposes.  For example, one URI producer might use a segment such as 
"name;v=1.1" to indicate a reference to version 1.1 of "name", whereas another 
might use a segment such as "name,1.1" to indicate the same.  Parameter types 
may be defined by scheme-specific semantics, but in most cases the syntax of a 
parameter is specific to the implementation of the URI's dereferencing 
algorithm.
{quote}
{noformat}
  reserved= gen-delims / sub-delims

  gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

  sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
  / "*" / "+" / "," / ";" / "="

...

  path  = path-abempty; begins with "/" or is empty
/ path-absolute   ; begins with "/" but not "//"
/ path-noscheme   ; begins with a non-colon segment
/ path-rootless   ; begins with a segment
/ path-empty  ; zero characters

  path-abempty  = *( "/" segment )
  path-absolute = "/" [ segment-nz *( "/" segment ) ]
  path-noscheme = segment-nz-nc *( "/" segment )
  path-rootless = segment-nz *( "/" segment )
  path-empty= 0
  segment   = *pchar
  segment-nz= 1*pchar
  segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
; non-zero-length segment without any colon ":"

  pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
{noformat}

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904731#comment-16904731
 ] 

Masatake Iwasaki edited comment on HDFS-14423 at 8/11/19 8:28 PM:
--

The reason why [WebHdfs file path gets truncated when having semicolon (\;) 
inside|HDFS-13176] is that Jersey treats the part after ";" as parameters. I 
think using URLEncoder/URLDecoder is not the right way for HDFS-13176. I'm 
going to try refatctoring here.


was (Author: iwasakims):
The reason why WebHdfs file path gets truncated when having semicolon (\;) 
inside is that Jersey treats the part after ";" as parameters. I think using 
URLEncoder/URLDecoder is not the right way for HDFS-13176. I'm going to try 
refatctoring here.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904731#comment-16904731
 ] 

Masatake Iwasaki edited comment on HDFS-14423 at 8/11/19 8:11 PM:
--

The reason why WebHdfs file path gets truncated when having semicolon (\;) 
inside is that Jersey treats the part after ";" as parameters. I think using 
URLEncoder/URLDecoder is not the right way for HDFS-13176. I'm going to try 
refatctoring here.


was (Author: iwasakims):
The reason why WebHdfs file path gets truncated when having semicolon (;) 
inside is that Jersey treats the part after ";" as parameters. I think using 
URLEncoder/URLDecoder is not the right way for HDFS-13176. I'm going to try 
refatctoring here.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904731#comment-16904731
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

The reason why WebHdfs file path gets truncated when having semicolon (;) 
inside is that Jersey treats the part after ";" as parameters. I think using 
URLEncoder/URLDecoder is not the right way for HDFS-13176. I'm going to try 
refatctoring here.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904728#comment-16904728
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

java.net.URLEncoder converts "\+" into " " (since java.net.URLEncoder converts 
" " into "\+"). The code path of {{open}} seems to convert "a+b" into "a b" 
while {{mkdir}} does not.

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-10 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904531#comment-16904531
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

Thanks [~jojochuang]. While the patch looks good, the issued reported as "plus 
(+ ) characters get turned into spaces" still seems to be exists. I'm looking 
into the cause.
{noformat}
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz 
webhdfs://localhost/%
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz 
webhdfs://localhost/a+b
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -mkdir 
webhdfs://localhost/c+d
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls webhdfs://localhost/
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hadoop fs -put 'e+f.txt' 
'webhdfs://localhost/'
put: rename `webhdfs://localhost/e+f.txt._COPYING_' to 
`webhdfs://localhost/e+f.txt': Input/output error
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls /
Found 6 items
-rw-r--r--   1 iwasakims supergroup  0 2019-08-09 23:35 /%
-rw-r--r--   1 iwasakims supergroup  0 2019-08-09 23:50 /a b
drwxr-xr-x   - iwasakims supergroup  0 2019-08-09 23:37 /c+d
-rw-r--r--   1 iwasakims supergroup  5 2019-08-09 23:52 /e 
f.txt._COPYING_
drwxrwxrwx   - iwasakims supergroup  0 2019-06-14 13:09 /tmp
drwxr-xr-x   - iwasakims supergroup  0 2019-06-14 13:10 /user
{noformat}

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-07 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901906#comment-16901906
 ] 

Masatake Iwasaki commented on HDFS-14707:
-

I manually tested the modified httpfs.sh using downloaded hadoop-2.9.2.tar.gz 
with debug logging enabled in httpfs-log4j.properties.

Native library was not loaded without the fix.
{noformat}
2019-08-07 17:24:51,001 DEBUG NativeCodeLoader [][:]  Trying to load the 
custom-built native-hadoop library...
2019-08-07 17:24:51,001 DEBUG NativeCodeLoader [][:]  Failed to load 
native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in 
java.library.path
2019-08-07 17:24:51,001 DEBUG NativeCodeLoader [][:]  
java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-08-07 17:24:51,002  WARN NativeCodeLoader [][:]  Unable to load 
native-hadoop library for your platform... using builtin-java classes where 
applicable
{noformat}

Native library was loaded with the patch.
{noformat}
2019-08-07 17:25:45,622 DEBUG NativeCodeLoader [][:]  Trying to load the 
custom-built native-hadoop library...
2019-08-07 17:25:45,623 DEBUG NativeCodeLoader [][:]  Loaded the native-hadoop 
library
{noformat}

>  Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2
> 
>
> Key: HDFS-14707
> URL: https://issues.apache.org/jira/browse/HDFS-14707
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14707-branch-2.001.patch
>
>
> Currently HTTPFS does not load hadoop native library since java.library.path 
> is not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-06 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14707:

Status: Patch Available  (was: Open)

>  Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2
> 
>
> Key: HDFS-14707
> URL: https://issues.apache.org/jira/browse/HDFS-14707
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14707-branch-2.001.patch
>
>
> Currently HTTPFS does not load hadoop native library since java.library.path 
> is not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-06 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14707:

Attachment: HDFS-14707-branch-2.001.patch

>  Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2
> 
>
> Key: HDFS-14707
> URL: https://issues.apache.org/jira/browse/HDFS-14707
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-14707-branch-2.001.patch
>
>
> Currently HTTPFS does not load hadoop native library since java.library.path 
> is not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-06 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901616#comment-16901616
 ] 

Masatake Iwasaki commented on HDFS-14707:
-

Same as done for KMS in HADOOP-11329.

>  Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2
> 
>
> Key: HDFS-14707
> URL: https://issues.apache.org/jira/browse/HDFS-14707
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> Currently HTTPFS does not load hadoop native library since java.library.path 
> is not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Created] (HDFS-14707) Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2

2019-08-06 Thread Masatake Iwasaki (JIRA)
Masatake Iwasaki created HDFS-14707:
---

 Summary:  Add JAVA_LIBRARY_PATH to HTTPFS startup options in 
branch-2
 Key: HDFS-14707
 URL: https://issues.apache.org/jira/browse/HDFS-14707
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: httpfs
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


Currently HTTPFS does not load hadoop native library since java.library.path is 
not set on Tomcat startup.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-07-28 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

Attachment: HDFS-14135-branch-2.013.patch

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135-branch-2.013.patch, HDFS-14135-branch-3.2.013.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch, HDFS-14135.013.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-07-26 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

Attachment: HDFS-14135-branch-3.2.013.patch

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135-branch-3.2.013.patch, HDFS-14135.009.patch, HDFS-14135.010.patch, 
> HDFS-14135.011.patch, HDFS-14135.012.patch, HDFS-14135.013.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-07-26 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894193#comment-16894193
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

Thanks [~xkrogen] for taking care. I'm going to backport the patch for older 
branch.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch, HDFS-14135.013.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-07-26 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.1.3
   3.2.1
   3.3.0
   2.10.0
   Status: Resolved  (was: Patch Available)

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch, HDFS-14135.013.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-07-25 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893232#comment-16893232
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

Thanks [~elgoiri]. I'm committing this.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch, HDFS-14135.013.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14640) [Dynamometer] Fix TestDynamometerInfra failures

2019-07-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882968#comment-16882968
 ] 

Masatake Iwasaki commented on HDFS-14640:
-

+1. The patch fixed test timeout on my CentOS VM.

> [Dynamometer] Fix TestDynamometerInfra failures
> ---
>
> Key: HDFS-14640
> URL: https://issues.apache.org/jira/browse/HDFS-14640
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test, tools
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14640.000.patch
>
>
> I've been seeing Jenkins reporting some failures of the 
> {{TestDynamometerInfra}} test (basically a big integration test). It seems 
> like it's timing out after 15 minutes.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14466) FileSystem.listLocatedStatus for path including '=' encodes it and returns FileNotFoundException

2019-07-11 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882760#comment-16882760
 ] 

Masatake Iwasaki commented on HDFS-14466:
-

Thanks [~aajisaka]. I updated the patch to add only the test and addressed 
checkstyle error. Attached as 002.

> FileSystem.listLocatedStatus for path including '=' encodes it and returns 
> FileNotFoundException
> 
>
> Key: HDFS-14466
> URL: https://issues.apache.org/jira/browse/HDFS-14466
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs, webhdfs
>Affects Versions: 3.2.0
>Reporter: Yuya Ebihara
>Assignee: Masatake Iwasaki
>Priority: Minor
>  Labels: webhdfs
> Attachments: HADOOP-16258.001.patch, HDFS-14466.002.patch
>
>
> Recently, we upgraded hadoop library from 2.7.7 to 3.2.0. This issue happens 
> after the update. When we call FileSystem.listLocatedStatus with location 
> 'webhdfs://hadoop-master:50070/user/hive/warehouse/test_part/dt=1', the 
> internal calls are
>  * 2.7.7 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx|http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx%27,]
>  * 3.2.0 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt%253D1?op=LISTSTATUS=xxx]'
> As a result, it returns RemoteException with FileNotFoundException.
> {code:java}
> {"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File
>  /user/hive/warehouse/test_part/dt%3D1 does not exist."}}
> {code}
> Could you please tell me whether it's a bug and the way to avoid it?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14466) FileSystem.listLocatedStatus for path including '=' encodes it and returns FileNotFoundException

2019-07-11 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14466:

Attachment: HDFS-14466.002.patch

> FileSystem.listLocatedStatus for path including '=' encodes it and returns 
> FileNotFoundException
> 
>
> Key: HDFS-14466
> URL: https://issues.apache.org/jira/browse/HDFS-14466
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs, webhdfs
>Affects Versions: 3.2.0
>Reporter: Yuya Ebihara
>Assignee: Masatake Iwasaki
>Priority: Minor
>  Labels: webhdfs
> Attachments: HADOOP-16258.001.patch, HDFS-14466.002.patch
>
>
> Recently, we upgraded hadoop library from 2.7.7 to 3.2.0. This issue happens 
> after the update. When we call FileSystem.listLocatedStatus with location 
> 'webhdfs://hadoop-master:50070/user/hive/warehouse/test_part/dt=1', the 
> internal calls are
>  * 2.7.7 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx|http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx%27,]
>  * 3.2.0 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt%253D1?op=LISTSTATUS=xxx]'
> As a result, it returns RemoteException with FileNotFoundException.
> {code:java}
> {"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File
>  /user/hive/warehouse/test_part/dt%3D1 does not exist."}}
> {code}
> Could you please tell me whether it's a bug and the way to avoid it?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14629) Property value Hard Coded in DNConf.java

2019-07-04 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14629:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.1.3
   3.2.1
   3.3.0
   2.10.0
   Status: Resolved  (was: Patch Available)

committed. Thanks for the contribution, [~hemanthboyina].

> Property value Hard Coded in DNConf.java
> 
>
> Key: HDFS-14629
> URL: https://issues.apache.org/jira/browse/HDFS-14629
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Trivial
> Fix For: 2.10.0, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14629.patch
>
>
> {code:java}
> DataNode.LOG.info("dfs.blockreport.initialDelay is "
> + "greater than or equal to" + "dfs.blockreport.intervalMsec."
> + " Setting initial delay to 0 msec:");{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14629) Property value Hard Coded in DNConf.java

2019-07-04 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878720#comment-16878720
 ] 

Masatake Iwasaki commented on HDFS-14629:
-

+1. committing this.

> Property value Hard Coded in DNConf.java
> 
>
> Key: HDFS-14629
> URL: https://issues.apache.org/jira/browse/HDFS-14629
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Trivial
> Attachments: HDFS-14629.patch
>
>
> {code:java}
> DataNode.LOG.info("dfs.blockreport.initialDelay is "
> + "greater than or equal to" + "dfs.blockreport.intervalMsec."
> + " Setting initial delay to 0 msec:");{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-06-26 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872958#comment-16872958
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

With the change of 013, test*ConnectTimeout (including 
testRedirectConnectTimeout and testTwoStepWriteConnectTimeout) are skipped on 
my local as expected on my local if I set CLIENTS_TO_CONSUME_BACKLOG < 
CONNECTION_BACKLOG to mimic "failed to fill up connection backlog." scenario.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch, HDFS-14135.013.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-06-20 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

Attachment: HDFS-14135.013.patch

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch, HDFS-14135.013.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-06-20 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869132#comment-16869132
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

Failed tests call {{consumeConnectionBacklog}} in another thread. I'm going to 
update the patch to cover this.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-06-20 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868977#comment-16868977
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

012 for addressing checkstyle errors and javac warnings.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-06-20 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

Attachment: HDFS-14135.012.patch

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch, 
> HDFS-14135.012.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-06-20 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868378#comment-16868378
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

I attached updated patch 011.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-06-20 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

Attachment: HDFS-14135.011.patch

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch, HDFS-14135.011.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-05-20 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844489#comment-16844489
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

I attached 010 using lambda.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-05-20 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

Attachment: HDFS-14135.010.patch

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch, HDFS-14135.010.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-05-20 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843845#comment-16843845
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

I attached 009 patch adding code to wait for filling-up of connection backlog.
{noformat}
+} catch (TimeoutException | InterruptedException e) {
+  fail("failed to fill up connection backlog.");
+}
{noformat}
Skipping test by throwing AssumptionViolatedException instead of {{fail}} would 
be an option since {{consumeConnectionBacklog}} seems to be assuming Linux env. 
The test could be unintentionally skipped in QA build is the downside. I chose 
{{fail}} here.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-05-20 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-14135:

Attachment: HDFS-14135.009.patch

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch, 
> HDFS-14135.009.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-05-14 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839883#comment-16839883
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

[~elgoiri] using reserved address did not work on test machine without default 
gateway. I'm trying to address race by fixing consumeConnectionBacklog.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk

2019-05-06 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833734#comment-16833734
 ] 

Masatake Iwasaki commented on HDFS-14135:
-

I got the same error on HDFS-14466. The cause seems to be race between 
{{consumeConnectionBacklog}} and tests as discussed here.

I guess that the reason why ServerSocket is used for "connect timed out" tests 
is to make sure the port is not accidentally listened by other tests or 
something. How about using [reserved 
address|https://en.wikipedia.org/wiki/Reserved_IP_addresses] like 192.0.2.1 as 
a target NN address rather than localhost? We don't need fill TCP backlog if 
the target address is always times out on connecting.

The "read timed out" tests still need ServerSocket and I think there is no 
timing issue.

> TestWebHdfsTimeouts Fails intermittently in trunk
> -
>
> Key: HDFS-14135
> URL: https://issues.apache.org/jira/browse/HDFS-14135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, 
> HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, 
> HDFS-14135-06.patch, HDFS-14135-07.patch, HDFS-14135-08.patch
>
>
> Reference to failure
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14466) FileSystem.listLocatedStatus for path including '=' encodes it and returns FileNotFoundException

2019-05-06 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833720#comment-16833720
 ] 

Masatake Iwasaki commented on HDFS-14466:
-

moved this to HDFS project.

> FileSystem.listLocatedStatus for path including '=' encodes it and returns 
> FileNotFoundException
> 
>
> Key: HDFS-14466
> URL: https://issues.apache.org/jira/browse/HDFS-14466
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs, webhdfs
>Affects Versions: 3.2.0
>Reporter: Yuya Ebihara
>Assignee: Masatake Iwasaki
>Priority: Minor
>  Labels: webhdfs
> Attachments: HADOOP-16258.001.patch
>
>
> Recently, we upgraded hadoop library from 2.7.7 to 3.2.0. This issue happens 
> after the update. When we call FileSystem.listLocatedStatus with location 
> 'webhdfs://hadoop-master:50070/user/hive/warehouse/test_part/dt=1', the 
> internal calls are
>  * 2.7.7 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx|http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx%27,]
>  * 3.2.0 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt%253D1?op=LISTSTATUS=xxx]'
> As a result, it returns RemoteException with FileNotFoundException.
> {code:java}
> {"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File
>  /user/hive/warehouse/test_part/dt%3D1 does not exist."}}
> {code}
> Could you please tell me whether it's a bug and the way to avoid it?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Moved] (HDFS-14466) FileSystem.listLocatedStatus for path including '=' encodes it and returns FileNotFoundException

2019-05-06 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki moved HADOOP-16258 to HDFS-14466:
--

Affects Version/s: (was: 3.2.0)
   3.2.0
  Component/s: (was: fs)
   webhdfs
   fs
  Key: HDFS-14466  (was: HADOOP-16258)
  Project: Hadoop HDFS  (was: Hadoop Common)

> FileSystem.listLocatedStatus for path including '=' encodes it and returns 
> FileNotFoundException
> 
>
> Key: HDFS-14466
> URL: https://issues.apache.org/jira/browse/HDFS-14466
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs, webhdfs
>Affects Versions: 3.2.0
>Reporter: Yuya Ebihara
>Assignee: Masatake Iwasaki
>Priority: Minor
>  Labels: webhdfs
> Attachments: HADOOP-16258.001.patch
>
>
> Recently, we upgraded hadoop library from 2.7.7 to 3.2.0. This issue happens 
> after the update. When we call FileSystem.listLocatedStatus with location 
> 'webhdfs://hadoop-master:50070/user/hive/warehouse/test_part/dt=1', the 
> internal calls are
>  * 2.7.7 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx|http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt=1?op=LISTSTATUS=xxx%27,]
>  * 3.2.0 
> [http://hadoop-master:50070/webhdfs/v1/user/hive/warehouse/test_part/dt%253D1?op=LISTSTATUS=xxx]'
> As a result, it returns RemoteException with FileNotFoundException.
> {code:java}
> {"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File
>  /user/hive/warehouse/test_part/dt%3D1 does not exist."}}
> {code}
> Could you please tell me whether it's a bug and the way to avoid it?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-14432) dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml

2019-04-17 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki resolved HDFS-14432.
-
   Resolution: Fixed
 Assignee: puleya7
Fix Version/s: 3.1.3
   3.2.1
   3.3.0
   3.0.4
   2.10.0

Thanks for the contribution, [~puleya7]. I committed this.

> dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml
> 
>
> Key: HDFS-14432
> URL: https://issues.apache.org/jira/browse/HDFS-14432
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Reporter: puleya7
>Assignee: puleya7
>Priority: Minor
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
>
> property "dfs.datanode.shared.file.descriptor.paths" appeared twice in 
> hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (after 
> HDFS-6007、2.5.0)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots

2019-02-27 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779517#comment-16779517
 ] 

Masatake Iwasaki commented on HDFS-14315:
-

Comments for TestRplicationSnapshot:

{code}
   public void testReplicationWithSnapshot() throws Exception {
-short fileRep = 1;
+short fileRep = 2;
 // Create file1, set its replication to 1
 DFSTestUtil.createFile(hdfs, file1, BLOCKSIZE, fileRep, seed);
 Map snapshotRepMap = new HashMap();
+FileStatus fileStatus = hdfs.getFileStatus(file1);
+long len = 0;
 // Change replication factor from 1 to 5. In the meanwhile, keep taking
 // snapshots for sub1
{code}
Changing initial value of fileRep makes following comments inconsistent.

{code}
+  for( BlockLocation locations : hdfs.getFileBlockLocations(fileStatus, 0, 
len) ) {
+//assertTrue(locations.getHosts().length == fileRep);
+assertEquals(fileRep, locations.getHosts().length);
{code}
The line commented out should be removed.

{code}
+// At this point in time, number of replicas (fileRep) is 5 and the latest 
snapshot
+// was taken when the replication factor was 4.
+
+//Change replication factor back to 3, but it's set to 4.
+hdfs.setReplication(file1, (short) 3);
{code}
I think there is confusion about terminology in the test. "fileRep" means 
replication factor set in inode. DistributedFileSystem#setReplication sets the 
given number to inode regardless of snapshots. 
INodeFile#getPreferredBlockReplication returns number of replicas adjusted 
considering snapshots. The number of replicas is called "blockRep" in the test.

{code}
+  //Wait for block replication
+  DFSTestUtil.waitForReplication(hdfs, file1, fileRep, 5000);
+
+  // Check number of replicas in datanodes, which should be 5.
+  len = fileStatus.getLen();
+  for( BlockLocation locations : hdfs.getFileBlockLocations(fileStatus, 0, 
len) ) {
+//assertTrue(locations.getHosts().length == fileRep);
+assertEquals(fileRep, locations.getHosts().length);
+  }
{code}
The check is redundant since DFSTestUtil#waitForReplication do the same thing. 
"should be 5" is wrong since fileRep is changing in the loop.

{code}
+//Wait for block invalitation in datanodes, but expect to receive the 
TimeoutException.
+try {
+  DFSTestUtil.waitForReplication(hdfs, file1, (short) 3, 1);
+} catch (Exception e) {
+  assertEquals(TimeoutException.class, e.getClass());
+}
{code}
TestReplication could be a reference if you want to check that there is no 
invalidation work without waiting 10s, while I think just 
checkSnapshotFileReplication is enough here.

> Add more detailed log message when decreasing replication factor < max in 
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch, HDFS-14315.001.patch
>
>
> When changing replication factor for a given file, the following 3 types of 
> logging appear in the namenode log, but more detailed message would be useful 
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to 
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the 
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO  
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) - 
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested 
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are 
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO  
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) - 
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested 
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.



--
This message was sent by Atlassian JIRA

[jira] [Commented] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots

2019-02-27 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779512#comment-16779512
 ] 

Masatake Iwasaki commented on HDFS-14315:
-

{code:java}
 if (oldBR != -1) {
-  if (oldBR > targetReplication) {
+  //File is likely in snapshots. Decreasing replication factor, but not 
the requested value
+  if (oldBR > targetReplication && targetReplication > replication) {
+FSDirectory.LOG.info("Decreasing replication from {} to {} for {} " +
+ "whereas requested value is {}. The file is 
likely in snapshots",
+ oldBR, targetReplication, iip.getPath(), 
replication, targetReplication);
+  //File is not in snapshots. Decreasing replication factor
+  } else if (oldBR > targetReplication){
 FSDirectory.LOG.info("Decreasing replication from {} to {} for {}",
  oldBR, targetReplication, iip.getPath());

{code}
The comment and log message should not mention snapshot. 
INodeFile#getPreferredBlockReplication would be fixed due to additional reason 
for adjusting number of replicas. My suggestion is just always logging given 
replication rather than adding conditionals like this.
{code:java}
 FSDirectory.LOG.info(
 "Decreasing replication from {} to {} (over {}) for {}",
 oldBR, targetReplication, replication, iip.getPath());
{code}

> Add more detailed log message when decreasing replication factor < max in 
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch, HDFS-14315.001.patch
>
>
> When changing replication factor for a given file, the following 3 types of 
> logging appear in the namenode log, but more detailed message would be useful 
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to 
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the 
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO  
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) - 
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested 
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are 
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO  
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) - 
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested 
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots

2019-02-25 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777553#comment-16777553
 ] 

Masatake Iwasaki commented on HDFS-14315:
-

[~daisuke.kobayashi] thanks for working on this.
{code:java}
import org.apache.hadoop.fs.*;
import org.apache.hadoop.hdfs.*;
{code}
We are usually avoiding to use wildcard import.
{code:java}
//Wait for block replication
try { Thread.sleep(1); } catch (InterruptedException ie) {}
{code}
Just sleeping makes test flaky or unnecessary long. 
DFSTestUtil#waitForReplication or similar should be used if you can.
{code:java}
  //File is in snapshots. Decreasing replication factor, but not the 
requested value 

 
  if (oldBR > targetReplication && targetReplication > replication &&
  iip.getLatestSnapshotId() != Snapshot.CURRENT_STATE_ID) {
FSDirectory.LOG.info("Decreasing replication from {} to {} for {}" +
 ". Requested value is {}, but {} is the maximum in 
snapshots",
 oldBR, targetReplication, iip.getPath(), 
replication, targetReplication);
{code}
This looks error prone for future change of 
INodeFile#getPreferredBlockReplication. FSDirAttrOp should not be aware of 
implementation details of snapshot as far as possible. Just adding requested 
replication facter to existing log would work for you?

> Add more detailed log message when decreasing replication factor < max in 
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch
>
>
> When changing replication factor for a given file, the following 3 types of 
> logging appear in the namenode log, but more detailed message would be useful 
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to 
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the 
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO  
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) - 
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested 
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are 
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO  
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) - 
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested 
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14240) blockReport test in NNThroughputBenchmark throws ArrayIndexOutOfBoundsException

2019-01-31 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757595#comment-16757595
 ] 

Masatake Iwasaki commented on HDFS-14240:
-

{quote}roblem is here:array datanodes's length is determined by args as 
"-datanodes" or "-threads" ,but dnIdx = dnInfo.getXferPort() is a random port.
{quote}
NNThroughputBenchmark registers fake datanode having port number based on idx 
if using in-process NameNode. The code path assuming the mode seems to be 
reached in the mode using real NameNode.

[~shenyinjie] if you can show the command line to reproduce the issue, it would 
be help.

[~RANith] I assigned you to this ticket.

 

> blockReport test in NNThroughputBenchmark throws 
> ArrayIndexOutOfBoundsException
> ---
>
> Key: HDFS-14240
> URL: https://issues.apache.org/jira/browse/HDFS-14240
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Shen Yinjie
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: screenshot-1.png
>
>
> When I run a blockReport test with NNThroughputBenchmark, 
> BlockReportStats.addBlocks() throws ArrayIndexOutOfBoundsException.
> digging the code:
> {code:java}
> for(DatanodeInfo dnInfo : loc.getLocations())
> { int dnIdx = dnInfo.getXferPort() - 1; 
> datanodes[dnIdx].addBlock(loc.getBlock().getLocalBlock());{code}
>  
> problem is here:array datanodes's length is determined by args as 
> "-datanodes" or "-threads" ,but dnIdx = dnInfo.getXferPort() is a random port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDFS-14240) blockReport test in NNThroughputBenchmark throws ArrayIndexOutOfBoundsException

2019-01-31 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki reassigned HDFS-14240:
---

Assignee: Ranith Sardar

> blockReport test in NNThroughputBenchmark throws 
> ArrayIndexOutOfBoundsException
> ---
>
> Key: HDFS-14240
> URL: https://issues.apache.org/jira/browse/HDFS-14240
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Shen Yinjie
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: screenshot-1.png
>
>
> When I run a blockReport test with NNThroughputBenchmark, 
> BlockReportStats.addBlocks() throws ArrayIndexOutOfBoundsException.
> digging the code:
> {code:java}
> for(DatanodeInfo dnInfo : loc.getLocations())
> { int dnIdx = dnInfo.getXferPort() - 1; 
> datanodes[dnIdx].addBlock(loc.getBlock().getLocalBlock());{code}
>  
> problem is here:array datanodes's length is determined by args as 
> "-datanodes" or "-threads" ,but dnIdx = dnInfo.getXferPort() is a random port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-12729) Document special paths in HDFS

2019-01-28 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-12729:

   Resolution: Fixed
Fix Version/s: 3.1.3
   2.9.3
   3.2.1
   2.8.6
   3.3.0
   3.0.4
   2.10.0
   Status: Resolved  (was: Patch Available)

> Document special paths in HDFS
> --
>
> Key: HDFS-12729
> URL: https://issues.apache.org/jira/browse/HDFS-12729
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: documentation
>Reporter: Chris Douglas
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-12729.001.patch
>
>
> Paths under {{/.reserved}} are handled specially by HDFS. The behavior of all 
> of these should be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-12729) Document special paths in HDFS

2019-01-25 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752642#comment-16752642
 ] 

Masatake Iwasaki commented on HDFS-12729:
-

Thanks [~ste...@apache.org]. I'm committing this.

> Document special paths in HDFS
> --
>
> Key: HDFS-12729
> URL: https://issues.apache.org/jira/browse/HDFS-12729
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: documentation
>Reporter: Chris Douglas
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-12729.001.patch
>
>
> Paths under {{/.reserved}} are handled specially by HDFS. The behavior of all 
> of these should be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-12729) Document special paths in HDFS

2019-01-22 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-12729:

Status: Patch Available  (was: Open)

> Document special paths in HDFS
> --
>
> Key: HDFS-12729
> URL: https://issues.apache.org/jira/browse/HDFS-12729
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: documentation
>Reporter: Chris Douglas
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-12729.001.patch
>
>
> Paths under {{/.reserved}} are handled specially by HDFS. The behavior of all 
> of these should be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-12729) Document special paths in HDFS

2019-01-22 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-12729:

Attachment: HDFS-12729.001.patch

> Document special paths in HDFS
> --
>
> Key: HDFS-12729
> URL: https://issues.apache.org/jira/browse/HDFS-12729
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: documentation
>Reporter: Chris Douglas
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-12729.001.patch
>
>
> Paths under {{/.reserved}} are handled specially by HDFS. The behavior of all 
> of these should be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-12729) Document special paths in HDFS

2019-01-22 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749233#comment-16749233
 ] 

Masatake Iwasaki commented on HDFS-12729:
-

Obviously docs of features using reserved names have description for the 
purpose.
* ExtendedAttributes.md
* HdfsSnapshots.md
* TransparentEncryption.md

There are brief mentions about .reserved and .snapshot in another docs.
* HdfsUserGuide.md (in "Upgrade and Rollback" section)
* FileSystemShell.md (in usage of -p option of cp)
* DistCp.md.vm (in usage of -p option)

If we add dedicated section about reserved names, HdfsDesign.md or 
HdfsUserGuide.md would be appropriate place.

> Document special paths in HDFS
> --
>
> Key: HDFS-12729
> URL: https://issues.apache.org/jira/browse/HDFS-12729
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: documentation
>Reporter: Chris Douglas
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HDFS-12729.001.patch
>
>
> Paths under {{/.reserved}} are handled specially by HDFS. The behavior of all 
> of these should be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDFS-12729) Document special paths in HDFS

2019-01-07 Thread Masatake Iwasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki reassigned HDFS-12729:
---

Assignee: Masatake Iwasaki

> Document special paths in HDFS
> --
>
> Key: HDFS-12729
> URL: https://issues.apache.org/jira/browse/HDFS-12729
> Project: Hadoop HDFS
>  Issue Type: Task
>  Components: documentation
>Reporter: Chris Douglas
>Assignee: Masatake Iwasaki
>Priority: Major
>
> Paths under {{/.reserved}} are handled specially by HDFS. The behavior of all 
> of these should be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-12654) APPEND API call is different in HTTPFS and NameNode REST

2018-03-06 Thread Masatake Iwasaki (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-12654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki resolved HDFS-12654.
-
Resolution: Not A Problem

I'm closing this as not-a-problem. Please reopen if you can reproduce the 500 
issue. We should update title on reopen. Thanks for pinging me, [~Sammi].

> APPEND API call is different in HTTPFS and NameNode REST
> 
>
> Key: HDFS-12654
> URL: https://issues.apache.org/jira/browse/HDFS-12654
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs, httpfs, namenode
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 3.0.0-beta1
>Reporter: Andras Czesznak
>Priority: Major
>
> The APPEND REST API call behaves differently in the NameNode REST and the 
> HTTPFS codes. The NameNode version creates the target file the new data being 
> appended to if it does not exist at the time of the call issued. The HTTPFS 
> version assumes the target file exists when APPEND is called and can append 
> only the new data but does not create the target file it doesn't exist.
> The two implementations should be standardized, preferably the HTTPFS version 
> should be modified to execute an implicit CREATE if the target file does not 
> exist.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-12654) APPEND API call is different in HTTPFS and NameNode REST

2018-01-29 Thread Masatake Iwasaki (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-12654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344419#comment-16344419
 ] 

Masatake Iwasaki commented on HDFS-12654:
-

Thanks for the info.

WebHDFS does not create a new file when append is requested. It returns 404. 
NamenodeWebHdfsMethods#chooseDataNode::
{noformat}
} else if (op == GetOpParam.Op.OPEN
|| op == GetOpParam.Op.GETFILECHECKSUM
|| op == PostOpParam.Op.APPEND) {
  //choose a datanode containing a replica
  final NamenodeProtocols np = getRPCServer(namenode);
  final HdfsFileStatus status = np.getFileInfo(path);
  if (status == null) {
throw new FileNotFoundException("File " + path + " not found.");
  }
{noformat}

The non-existent file seems to be crated by fluent-plugin-webhdfs. 
out_webhdfs.rb::
{noformat}
  def send_data(path, data)
if @append
  begin
@client.append(path, data)
  rescue WebHDFS::FileNotFoundError
@client.create(path, data)
  end
{noformat}

The issue stated in the ticket is that WebHDFS returns 404 but HttpFs returns 
500. I could not reproduce this.
{quote}
WebHDFS::ServerError means that the client (fluentd) receives HTTP response 
code 500 from HttpFs server. WebHDFS server returns 404 for such cases.
{quote}
 

> APPEND API call is different in HTTPFS and NameNode REST
> 
>
> Key: HDFS-12654
> URL: https://issues.apache.org/jira/browse/HDFS-12654
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs, httpfs, namenode
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 3.0.0-beta1
>Reporter: Andras Czesznak
>Priority: Major
>
> The APPEND REST API call behaves differently in the NameNode REST and the 
> HTTPFS codes. The NameNode version creates the target file the new data being 
> appended to if it does not exist at the time of the call issued. The HTTPFS 
> version assumes the target file exists when APPEND is called and can append 
> only the new data but does not create the target file it doesn't exist.
> The two implementations should be standardized, preferably the HTTPFS version 
> should be modified to execute an implicit CREATE if the target file does not 
> exist.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-12654) APPEND API call is different in HTTPFS and NameNode REST

2018-01-16 Thread Masatake Iwasaki (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-12654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327000#comment-16327000
 ] 

Masatake Iwasaki commented on HDFS-12654:
-

Hi [~Nuke], could you explain the way to reproduce the issue? I got the same 
result for webhdfs and httpfs when I tested this on 3.1.0-SNAPSHOT.

{noformat}
$ curl -X POST -T README.txt -L -i 
"http://localhost:9870/webhdfs/v1/tmp/README.txt?op=APPEND;
HTTP/1.1 100 Continue

HTTP/1.1 404 Not Found
Date: Tue, 16 Jan 2018 10:41:24 GMT
Cache-Control: no-cache
Expires: Tue, 16 Jan 2018 10:41:24 GMT
Date: Tue, 16 Jan 2018 10:41:24 GMT
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Content-Type: application/json
Transfer-Encoding: chunked

{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File
 /tmp/README.txt not found."}}[centos@ip-172-32-1-195 hadoop-3.1.0-SNAPSHOT]$
{noformat}

{noformat}
$ curl -X POST -i --header "Content-Type:application/octet-stream" 
--data-binary @README.txt 
'http://localhost:14000/webhdfs/v1/tmp/README.txt?op=APPEND=centos=true'
HTTP/1.1 100 Continue

HTTP/1.1 404 Not Found
Date: Tue, 16 Jan 2018 10:42:17 GMT
Cache-Control: no-cache
Expires: Tue, 16 Jan 2018 10:42:17 GMT
Date: Tue, 16 Jan 2018 10:42:17 GMT
Pragma: no-cache
Set-Cookie: 
hadoop.auth="u=centos=centos=simple-dt=1516135337720=ZDSoknG+/x/a6XnGLAVUyUBs6vE=";
 Path=/; HttpOnly
Content-Type: application/json
Transfer-Encoding: chunked

{"RemoteException":{"message":"Failed to append to non-existent file 
\/tmp\/README.txt for client 
127.0.0.1","exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException"}}
{noformat}
 

 

> APPEND API call is different in HTTPFS and NameNode REST
> 
>
> Key: HDFS-12654
> URL: https://issues.apache.org/jira/browse/HDFS-12654
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs, httpfs, namenode
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 3.0.0-beta1
>Reporter: Andras Czesznak
>Priority: Major
>
> The APPEND REST API call behaves differently in the NameNode REST and the 
> HTTPFS codes. The NameNode version creates the target file the new data being 
> appended to if it does not exist at the time of the call issued. The HTTPFS 
> version assumes the target file exists when APPEND is called and can append 
> only the new data but does not create the target file it doesn't exist.
> The two implementations should be standardized, preferably the HTTPFS version 
> should be modified to execute an implicit CREATE if the target file does not 
> exist.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-11995) HDFS Architecture documentation incorrectly describes writing to a local temporary file.

2017-06-19 Thread Masatake Iwasaki (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki resolved HDFS-11995.
-
   Resolution: Fixed
Fix Version/s: 2.8.2
   3.0.0-alpha4
   2.9.0

Committed this to branch-2.8.2 and above. Thanks, [~nandakumar131].

> HDFS Architecture documentation incorrectly describes writing to a local 
> temporary file.
> 
>
> Key: HDFS-11995
> URL: https://issues.apache.org/jira/browse/HDFS-11995
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.0.0-alpha3
>Reporter: Chris Nauroth
>Assignee: Nandakumar
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HDFS-11995.000.patch
>
>
> The HDFS Architecture documentation has a section titled "Staging" that 
> describes clients writing to a local temporary file first before interacting 
> with the NameNode to allocate file metadata.  This information is incorrect.  
> (Perhaps it was correct a long time ago, but it is no longer accurate with 
> respect to the current implementation.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-11995) HDFS Architecture documentation incorrectly describes writing to a local temporary file.

2017-06-19 Thread Masatake Iwasaki (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054913#comment-16054913
 ] 

Masatake Iwasaki commented on HDFS-11995:
-

+1, committing this.

If you say "local buffer in memory" rather than "local file", the statement 
might be not so wrong while the section name "staging" looks misleading.

> HDFS Architecture documentation incorrectly describes writing to a local 
> temporary file.
> 
>
> Key: HDFS-11995
> URL: https://issues.apache.org/jira/browse/HDFS-11995
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.0.0-alpha3
>Reporter: Chris Nauroth
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-11995.000.patch
>
>
> The HDFS Architecture documentation has a section titled "Staging" that 
> describes clients writing to a local temporary file first before interacting 
> with the NameNode to allocate file metadata.  This information is incorrect.  
> (Perhaps it was correct a long time ago, but it is no longer accurate with 
> respect to the current implementation.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-11583) Parent spans not initialized to NullScope for every DFSPacket

2017-05-29 Thread Masatake Iwasaki (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028408#comment-16028408
 ] 

Masatake Iwasaki commented on HDFS-11583:
-

I need +1 from another committer to commit this, pinging [~cmccabe] and 
[~stack] for review.

> Parent spans not initialized to NullScope for every DFSPacket
> -
>
> Key: HDFS-11583
> URL: https://issues.apache.org/jira/browse/HDFS-11583
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tracing
>Affects Versions: 2.7.1
>Reporter: Karan Mehta
>Assignee: Masatake Iwasaki
> Attachments: HDFS-11583-branch-2.7.001.patch, 
> HDFS-11583-branch-2.7.002.patch, HDFS-11583-branch-2.7.003.patch
>
>
> The issue was found while working with PHOENIX-3752.
> Each packet received by the {{run()}} method of {{DataStreamer}} class, uses 
> the {{parents}} field of the {{DFSPacket}} to create a new {{dataStreamer}} 
> span, which in turn creates a {{writeTo}} span as its child span. The parents 
> field is initialized when the packet is added to the {{dataQueue}} and the 
> value is initialized from the {{ThreadLocal}}. This is how HTrace handles 
> spans. 
> A {{TraceScope}} is created and initialized to {{NullScope}} before the loop 
> which runs till the point when the stream is closed. 
> Consider the following scenario, when the {{dataQueue}} contains multiple 
> packets, only the first of which has a tracing enabled. The scope is 
> initialized to the {{dataStreamer}} scope and a {{writeTo}} span is created 
> as its child, which gets closed once the packet is sent out to a remote 
> datanode. Before {{writeTo}} span is started, the {{dataStreamer}} scope is 
> detached. So calling the close method on it doesn't do anything at the end of 
> loop. 
> The second iteration will be using the stale value of the {{scope}} variable 
> with a DFSPacket on which tracing is not enabled. This results in generation 
> of an orphan {{writeTo}} spans which are being delivered to the 
> {{SpanReceiver}} as registered in the TraceFramework. This may result in 
> unlimited number of spans being generated and sent out to receiver. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-11765) Fix:Performance regression due to incorrect use of DataChecksum

2017-05-09 Thread Masatake Iwasaki (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004062#comment-16004062
 ] 

Masatake Iwasaki commented on HDFS-11765:
-

I will commit this shortly. Before that, I'm going to move this issue to Hadoop 
Common to which the fix belongs.

> Fix:Performance regression due to incorrect use of DataChecksum
> ---
>
> Key: HDFS-11765
> URL: https://issues.apache.org/jira/browse/HDFS-11765
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: native, performance
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: LiXin Ge
>Assignee: LiXin Ge
> Attachments: HDFS-11765.patch
>
>
> Recently I have upgraded my Hadoop version from 2.6 to 3.0, and I find that 
> the write performance decreased by 13%. After some days comparative analysis, 
> It's seems introduced by HADOOP-10865. 
> Since James Thomas have done the work that native checksum can run against 
> byte[] arrays instead of just against byte buffers, we may use native method 
> preferential because it runs faster than others.
> [~szetszwo] and [~iwasakims] could you take a look at this to see if  it make 
> bad effect on your benchmark test? [~tlipcon] could you help to see if I have 
> make mistakes in this patch?
> thanks!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-11765) Fix:Performance regression due to incorrect use of DataChecksum

2017-05-09 Thread Masatake Iwasaki (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002334#comment-16002334
 ] 

Masatake Iwasaki commented on HDFS-11765:
-

+1. I saw no relevant test failures in hadoop-hdfs. 

bq. BTW, the default checksum type is CRC32C in hadoop, which means HDFS will 
use PureJava algorithm when we didn't let it use native.

Sure. Discussion about further optimization should be done in another JIRA. 

> Fix:Performance regression due to incorrect use of DataChecksum
> ---
>
> Key: HDFS-11765
> URL: https://issues.apache.org/jira/browse/HDFS-11765
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: native, performance
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: LiXin Ge
>Assignee: LiXin Ge
> Attachments: HDFS-11765.patch
>
>
> Recently I have upgraded my Hadoop version from 2.6 to 3.0, and I find that 
> the write performance decreased by 13%. After some days comparative analysis, 
> It's seems introduced by HADOOP-10865. 
> Since James Thomas have done the work that native checksum can run against 
> byte[] arrays instead of just against byte buffers, we may use native method 
> preferential because it runs faster than others.
> [~szetszwo] and [~iwasakims] could you take a look at this to see if  it make 
> bad effect on your benchmark test? [~tlipcon] could you help to see if I have 
> make mistakes in this patch?
> thanks!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-11765) Fix:Performance regression due to incorrect use of DataChecksum

2017-05-08 Thread Masatake Iwasaki (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001850#comment-16001850
 ] 

Masatake Iwasaki commented on HDFS-11765:
-

The patch lgtm too. I'm running HDFS tests with the patch applied. +1 if there 
is no failures except for known flaky ones.

bq. I find that the write performance decreased by 13%

[~GeLiXin], can you share how you did the comparison? It would be nice if we 
can reproduce the issue by TestDFSIO or similar.

BlockReceiver seems to be relevant caller of {{DataChecksum#BlockReceiver}} 
with non-direct buffer. Based on the result of Crc32PerformanceTest, 
NativeCrc32 is faster than CRC32 if bytes-per-checksum <= 128 on my environment.

> Fix:Performance regression due to incorrect use of DataChecksum
> ---
>
> Key: HDFS-11765
> URL: https://issues.apache.org/jira/browse/HDFS-11765
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: native, performance
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: LiXin Ge
>Assignee: LiXin Ge
> Attachments: HDFS-11765.patch
>
>
> Recently I have upgraded my Hadoop version from 2.6 to 3.0, and I find that 
> the write performance decreased by 13%. After some days comparative analysis, 
> It's seems introduced by HADOOP-10865. 
> Since James Thomas have done the work that native checksum can run against 
> byte[] arrays instead of just against byte buffers, we may use native method 
> preferential because it runs faster than others.
> [~szetszwo] and [~iwasakims] could you take a look at this to see if  it make 
> bad effect on your benchmark test? [~tlipcon] could you help to see if I have 
> make mistakes in this patch?
> thanks!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



  1   2   3   4   5   6   >