[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-16 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953245#comment-16953245
 ] 

Wei-Chiu Chuang commented on HADOOP-16152:
--

On a second thought, Hadoop shouldn't be blocked by Tez so I'll go ahead commit 
this one.
But I'll comment on TEZ-4083 that without the change it won't be able to 
support Hadoop 3.3.0 and more likely lower releases because I will have to 
cherry pick this commit into lower branches.

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, 
> HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, 
> HADOOP-16152.006.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16655) Use http when fetching tomcat tarball for branch-2

2019-10-15 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952403#comment-16952403
 ] 

Wei-Chiu Chuang commented on HADOOP-16655:
--

LGTM +1

http://www.apache.org/dev/release-download-pages.html states cchecksums, 
detached signatures and public key should use HTTPS but it doesn't state use 
http for artifacts. So not the best, but that's probably ok.

> Use http when fetching tomcat tarball for branch-2
> --
>
> Key: HADOOP-16655
> URL: https://issues.apache.org/jira/browse/HADOOP-16655
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: HADOOP-16655-branch-2.001.patch, 
> HADOOP-16655-branch-2.002.patch
>
>
> Hit this error when building via docker:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project 
> hadoop-kms: An Ant BuildException has occured: 
> javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
> [ERROR] around Ant part ... skipexisting="true" verbose="true" 
> src="https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.43/bin/apache-tomcat-8.5.43.tar.gz"/>...
>  @ 5:183 in 
> /build/source/hadoop-common-project/hadoop-kms/target/antrun/build-main.xml
> [ERROR] -> [Help 1] {noformat}
> Seems this is caused by HADOOP-16323 which fetches via https.
> This should only be an issue in branch-2 since this was removed for KMS in 
> HADOOP-13597, and httpfs in HDFS-10860
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16655) Use http when fetching tomcat tarball for branch-2

2019-10-15 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952354#comment-16952354
 ] 

Wei-Chiu Chuang commented on HADOOP-16655:
--

I think it's a problem with JDK7?

This is what i have when i use JDK7 to build:
{noformat}
JAVA_HOME=`/usr/libexec/java_home -v 1.7` mvn -Dhttps.protocols=TLSv1.2  
-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 clean install 
-DskipTests -Pdist -Dmaven.javadoc.skip=true
{noformat}


> Use http when fetching tomcat tarball for branch-2
> --
>
> Key: HADOOP-16655
> URL: https://issues.apache.org/jira/browse/HADOOP-16655
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: HADOOP-16655-branch-2.001.patch
>
>
> Hit this error when building via docker:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project 
> hadoop-kms: An Ant BuildException has occured: 
> javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
> [ERROR] around Ant part ... skipexisting="true" verbose="true" 
> src="https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.43/bin/apache-tomcat-8.5.43.tar.gz"/>...
>  @ 5:183 in 
> /build/source/hadoop-common-project/hadoop-kms/target/antrun/build-main.xml
> [ERROR] -> [Help 1] {noformat}
> Seems this is caused by HADOOP-16323 which fetches via https.
> This should only be an issue in branch-2 since this was removed for KMS in 
> HADOOP-13597, and httpfs in HDFS-10860
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-15 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15169:
-
Fix Version/s: 3.3.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

Pushed to trunk, thanks [~brahmareddy] for the initial patch, and [~aajisaka] 
and [~xyao] for the review and comments!

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.003.patch, HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16643) Update netty4 to the latest 4.1.42

2019-10-15 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16643:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks [~leosun08]!

> Update netty4 to the latest 4.1.42
> --
>
> Key: HADOOP-16643
> URL: https://issues.apache.org/jira/browse/HADOOP-16643
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16643.001.patch
>
>
> The latest netty is out. Let's update it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16643) Update netty4 to the latest 4.1.42

2019-10-15 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16643:
-
Fix Version/s: 3.3.0

> Update netty4 to the latest 4.1.42
> --
>
> Key: HADOOP-16643
> URL: https://issues.apache.org/jira/browse/HADOOP-16643
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16643.001.patch
>
>
> The latest netty is out. Let's update it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-14 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951436#comment-16951436
 ] 

Wei-Chiu Chuang commented on HADOOP-15169:
--

It is fine in the current Jetty version and Hadoop 3.3.0 where the default 
excluded protocols in Jetty and default enabled protocols in Hadoop don't 
overlap. The effect is the same where the order is reversed or not.

If one day we update to a future Jetty version that excludes more protocols and 
some of which are permitted by Hadoop by default, I would like them not to be 
enabled (Jetty's exclude list takes precedence over include list) by default, 
unless user consciously update the hadoop.ssl.enabled.protocols configuration.

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.003.patch, HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-14 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15169:
-
Attachment: HADOOP-15169.003.patch

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.003.patch, HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16643) Update netty4 to the latest 4.1.42

2019-10-11 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949833#comment-16949833
 ] 

Wei-Chiu Chuang commented on HADOOP-16643:
--

Everything looks good so far. Doesn't seem to break downstreams. Let me know if 
there are any objections otherwise I'll commit this to trunk next week. I'm a 
little hesitate to cherry pick the commit to lower branches, but I probably 
will.

> Update netty4 to the latest 4.1.42
> --
>
> Key: HADOOP-16643
> URL: https://issues.apache.org/jira/browse/HADOOP-16643
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16643.001.patch
>
>
> The latest netty is out. Let's update it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL

2019-10-11 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949827#comment-16949827
 ] 

Wei-Chiu Chuang commented on HADOOP-13836:
--

I think this work is superseded by HADOOP-15977 where Daryn has made good 
progress.

> Securing Hadoop RPC using SSL
> -
>
> Key: HADOOP-13836
> URL: https://issues.apache.org/jira/browse/HADOOP-13836
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: ipc
>Reporter: kartheek muthyala
>Assignee: kartheek muthyala
>Priority: Major
> Attachments: HADOOP-13836-v2.patch, HADOOP-13836-v3.patch, 
> HADOOP-13836-v4.patch, HADOOP-13836.patch, Secure IPC OSS Proposal-1.pdf, 
> SecureIPC Performance Analysis-OSS.pdf
>
>
> Today, RPC connections in Hadoop are encrypted using Simple Authentication & 
> Security Layer (SASL), with the Kerberos ticket based authentication or 
> Digest-md5 checksum based authentication protocols. This proposal is about 
> enhancing this cipher suite with SSL/TLS based encryption and authentication. 
> SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that 
> provides data security and integrity across two different end points in a 
> network. This protocol has made its way to a number of applications such as 
> web browsing, email, internet faxing, messaging, VOIP etc. And supporting 
> this cipher suite at the core of Hadoop would give a good synergy with the 
> applications on top and also bolster industry adoption of Hadoop.
> The Server and Client code in Hadoop IPC should support the following modes 
> of communication
> 1.Plain 
> 2. SASL encryption with an underlying authentication
> 3. SSL based encryption and authentication (x509 certificate)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-11 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949824#comment-16949824
 ] 

Wei-Chiu Chuang edited comment on HADOOP-15169 at 10/11/19 10:49 PM:
-

Thanks [~xyao]!

Good idea. I'll update the patch accordingly.

However, Jetty won't let me enable SSLv2Hello only. I must also enable another 
protocol otherwise it fails right away. I think that's by design. So this 
doesn't really need a test case.


was (Author: jojochuang):
Thanks [~xyao]!

Good idea. I'll update the patch accordingly.

If Jetty won't let me enable SSLv2Hello only. I must also enable another 
protocol otherwise it fails right away. I think that's by design. So this 
doesn't really need a test case.

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-11 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949824#comment-16949824
 ] 

Wei-Chiu Chuang commented on HADOOP-15169:
--

Thanks [~xyao]!

Good idea. I'll update the patch accordingly.

If Jetty won't let me enable SSLv2Hello only. I must also enable another 
protocol otherwise it fails right away. I think that's by design. So this 
doesn't really need a test case.

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-11 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949492#comment-16949492
 ] 

Wei-Chiu Chuang commented on HADOOP-16152:
--

This patch breaks the Tez version we use. I think we need TEZ-4083 too.

{noformat}
2019-10-08 21:46:59,220 [INFO] [main] |service.AbstractService|: Service 
org.apache.tez.dag.app.DAGAppMaster failed in state STARTED
org.apache.hadoop.service.ServiceStateException: 
java.lang.NoClassDefFoundError: org/eclipse/jetty/util/ClassVisibilityChecker
at 
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at 
org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1968)
at 
org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:2035)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2682)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at 
org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2678)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2484)
Caused by: java.lang.NoClassDefFoundError: 
org/eclipse/jetty/util/ClassVisibilityChecker
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:509)
at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:515)
{noformat}

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, 
> HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, 
> HADOOP-16152.006.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-09 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948096#comment-16948096
 ] 

Wei-Chiu Chuang commented on HADOOP-15169:
--

[~brahmareddy] After Jetty 9.2.4.v20141103 (which is the case after Hadoop 
3.0), SSLv2Hello is excluded by default, and we have to explicitly reset 
excluded protocols otherwise your change won't take effect.

I am attaching an updated patch with a test, which correctly add supported 
protocols specified. Please take a look.

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-09 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948096#comment-16948096
 ] 

Wei-Chiu Chuang edited comment on HADOOP-15169 at 10/10/19 12:41 AM:
-

[~brahmareddy] After Jetty 9.2.4.v20141103 (which is the case after Hadoop 
3.0), SSLv2Hello is excluded by default, and we have to explicitly reset 
excluded protocols otherwise your change won't take effect.

I am attaching an updated patch with a test, which correctly add supported 
protocols specified. Please take a look.

Note also that an optional include list of protocols can potentially disable 
vulnerable protocols in the future as well.


was (Author: jojochuang):
[~brahmareddy] After Jetty 9.2.4.v20141103 (which is the case after Hadoop 
3.0), SSLv2Hello is excluded by default, and we have to explicitly reset 
excluded protocols otherwise your change won't take effect.

I am attaching an updated patch with a test, which correctly add supported 
protocols specified. Please take a look.

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-10-09 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15169:
-
Attachment: HADOOP-15169.002.patch

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.002.patch, 
> HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16647) Support OpenSSL 1.1.1 LTS

2019-10-09 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16647:
-
 Component/s: security
Target Version/s: 2.10.0, 3.3.0

At very least (assuming Hadoop 3 is compatible with OpenSSL 1.1.1 and no dev 
work is required) we should backport HADOOP-14597 to branch-2

> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
>  Issue Type: Task
>  Components: security
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> See Hadoop user mailing list 
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux 
> distros, but it's not clear to me if Linux distros are going support 
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the 
> openssl version supported. File this jira to test/document/fix bugs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16647) Support OpenSSL 1.1.1 LTS

2019-10-09 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16647:
-
Summary: Support OpenSSL 1.1.1 LTS  (was: Support OpenSSL 1.1.0 LTS)

> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> See Hadoop user mailing list 
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux 
> distros, but it's not clear to me if Linux distros are going support 
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the 
> openssl version supported. File this jira to test/document/fix bugs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Created] (HADOOP-16647) Support OpenSSL 1.1.0 LTS

2019-10-09 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HADOOP-16647:


 Summary: Support OpenSSL 1.1.0 LTS
 Key: HADOOP-16647
 URL: https://issues.apache.org/jira/browse/HADOOP-16647
 Project: Hadoop Common
  Issue Type: Task
Reporter: Wei-Chiu Chuang


See Hadoop user mailing list 
http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E

Hadoop 2 supports OpenSSL 1.0.2.
Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.

Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
* 1.1.0 is EOL 2019/09/11
* 1.0.2 EOL 2019/12/31
* 1.1.1 is EOL 2023/09/11 (LTS)

Many Hadoop installation relies on the OpenSSL package provided by Linux 
distros, but it's not clear to me if Linux distros are going support 
1.1.0/1.0.2 beyond this date.

We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the 
openssl version supported. File this jira to test/document/fix bugs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-09 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947827#comment-16947827
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

Ah i stand corrected. There are a few test methods that failed in that test.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-09 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947820#comment-16947820
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

No. I only updated zookeeper version on top your patch. Tested on top of Hadoop 
trunk branch.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-08 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947313#comment-16947313
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

Once I updated zookeeper to 3.5.5 too, curator tests passed. I'll subject this 
patch to more tests.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16643) Update netty4 to the latest 4.1.42

2019-10-08 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947228#comment-16947228
 ] 

Wei-Chiu Chuang commented on HADOOP-16643:
--

This patch passes Cloudera CDP L0 tests. I'll run more tests.

> Update netty4 to the latest 4.1.42
> --
>
> Key: HADOOP-16643
> URL: https://issues.apache.org/jira/browse/HADOOP-16643
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16643.001.patch
>
>
> The latest netty is out. Let's update it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16491) Upgrade jetty version to 9.3.27

2019-10-08 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16491:
-
Fix Version/s: 3.2.2
   3.1.4

> Upgrade jetty version to 9.3.27
> ---
>
> Key: HADOOP-16491
> URL: https://issues.apache.org/jira/browse/HADOOP-16491
> Project: Hadoop Common
>  Issue Type: Task
>Affects Versions: 3.2.0
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-16491-001.patch
>
>
> The current jetty version (9.3.24) has few CVEs (Ref: 
> [https://www.cvedetails.com/version/272598/Eclipse-Jetty-9.3.24.html]). It 
> would be a good idea to upgrade jetty to 9.3.27 (which is the latest version 
> as of today 08/05/2019).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16643) Update netty4 to the latest 4.1.42

2019-10-08 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946985#comment-16946985
 ] 

Wei-Chiu Chuang commented on HADOOP-16643:
--

Thanks for the patch, [~leosun08].

We internally use netty 4.1, and upstream trunk uses netty 4.0. Updating for my 
internal branch should be trivial, but for upstream Hadoop, a minor release 
update like this usually brings more changes than it looks like. I'll apply 
this patch to our internal branch and test it out. Will report back.

> Update netty4 to the latest 4.1.42
> --
>
> Key: HADOOP-16643
> URL: https://issues.apache.org/jira/browse/HADOOP-16643
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16643.001.patch
>
>
> The latest netty is out. Let's update it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation

2019-10-08 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946911#comment-16946911
 ] 

Wei-Chiu Chuang commented on HADOOP-15616:
--

This is in trunk, right? We should update the Fix Version to 3.3.0, so that RM 
doesn't skip this one.

> Incorporate Tencent Cloud COS File System Implementation
> 
>
> Key: HADOOP-15616
> URL: https://issues.apache.org/jira/browse/HADOOP-15616
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs/cos
>Reporter: Junping Du
>Assignee: YangY
>Priority: Major
> Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, 
> HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, 
> HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, 
> HADOOP-15616.009.patch, HADOOP-15616.010.patch, HADOOP-15616.011.patch, 
> Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf
>
>
> Tencent cloud is top 2 cloud vendors in China market and the object store COS 
> ([https://intl.cloud.tencent.com/product/cos]) is widely used among China’s 
> cloud users but now it is hard for hadoop user to access data laid on COS 
> storage as no native support for COS in Hadoop.
> This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just 
> like what we do before for S3, ADL, OSS, etc. With simple configuration, 
> Hadoop applications can read/write data from COS without any code change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Created] (HADOOP-16643) Upate netty4 to the latest 4.1.42

2019-10-08 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HADOOP-16643:


 Summary: Upate netty4 to the latest 4.1.42
 Key: HADOOP-16643
 URL: https://issues.apache.org/jira/browse/HADOOP-16643
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


The latest netty is out. Let's update it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16619) Upgrade jackson and jackson-databind to 2.9.10

2019-10-08 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946702#comment-16946702
 ] 

Wei-Chiu Chuang commented on HADOOP-16619:
--

We should backport it to lower branches.

> Upgrade jackson and jackson-databind to 2.9.10
> --
>
> Key: HADOOP-16619
> URL: https://issues.apache.org/jira/browse/HADOOP-16619
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
>
> Two more CVEs (CVE-2019-16335 and CVE-2019-14540) are addressed in 
> jackson-databind 2.9.10.
> For details see Jackson Release 2.9.10 [release 
> notes|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16491) Upgrade jetty version to 9.3.27

2019-10-08 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946701#comment-16946701
 ] 

Wei-Chiu Chuang commented on HADOOP-16491:
--

We should cherry pick this into lower branches.

> Upgrade jetty version to 9.3.27
> ---
>
> Key: HADOOP-16491
> URL: https://issues.apache.org/jira/browse/HADOOP-16491
> Project: Hadoop Common
>  Issue Type: Task
>Affects Versions: 3.2.0
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16491-001.patch
>
>
> The current jetty version (9.3.24) has few CVEs (Ref: 
> [https://www.cvedetails.com/version/272598/Eclipse-Jetty-9.3.24.html]). It 
> would be a good idea to upgrade jetty to 9.3.27 (which is the latest version 
> as of today 08/05/2019).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-07 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946430#comment-16946430
 ] 

Wei-Chiu Chuang commented on HADOOP-16152:
--

The compilation error doesn't look related. See HDFS-14900

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, 
> HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, 
> HADOOP-16152.006.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-07 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946154#comment-16946154
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

Thank you, Norbert. Our unit test doesn't actually run any useful tests when 
you update only the pom.xml file.

We probably want to update to zookeeper 3.5.6, assuming it doesn't have 
performance regression (we have heavy dependency on zookeeper for KMS and 
Router Based Federation), and backward compat. We can also gradually get rid of 
netty3.

[~inigoiri] FYI, like I mentioned, RBF have heavy dependency on ZK, so you 
should care about it.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-05 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945131#comment-16945131
 ] 

Wei-Chiu Chuang commented on HADOOP-16152:
--

Some of the failures are related to HADOOP-16579 which I reverted.


> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, 
> HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.005.patch, 
> HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-05 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16579:
-
Fix Version/s: (was: 3.3.0)

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Reopened] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-05 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang reopened HADOOP-16579:
--

It's causing unit test failures. I'm going to revert this change.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
> Fix For: 3.3.0
>
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-04 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16579:
-
Fix Version/s: 3.3.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

Thanks [~nkalmar].
The patch passes our internal basic tests. Merged the PR and we'll work on 
further integration tests. Will come back to this if something wrong happens.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
> Fix For: 3.3.0
>
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-10-04 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944587#comment-16944587
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

+1 will commit soon.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16127) In ipc.Client, put a new connection could happen after stop

2019-10-03 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16127:
-
Fix Version/s: 3.2.2
   3.1.4

> In ipc.Client, put a new connection could happen after stop
> ---
>
> Key: HADOOP-16127
> URL: https://issues.apache.org/jira/browse/HADOOP-16127
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Reporter: Tsz-wo Sze
>Assignee: Tsz-wo Sze
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: c16127_20190219.patch, c16127_20190220.patch, 
> c16127_20190225.patch
>
>
> In getConnection(..), running can be initially true but becomes false before 
> putIfAbsent.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-12282) Connection thread's name should be updated after address changing is detected

2019-10-03 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-12282:
-
Fix Version/s: 3.2.2
   3.1.4

> Connection thread's name should be updated after address changing is detected
> -
>
> Key: HADOOP-12282
> URL: https://issues.apache.org/jira/browse/HADOOP-12282
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Reporter: zhouyingchao
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-12282-001.patch, HADOOP-12282.002.patch
>
>
> In a hadoop hdfs cluster, I changed the standby Namenode's ip address (the 
> hostname is not changed and the routing tables are updated). After the 
> change, the cluster is running as normal.
>  However, I found that the debug message of datanode's IPC still prints the 
> original ip address. By looking into the implementation, it turns out that 
> the original address is used as the thread's name. I think the thread's name 
> should be changed if the address change is detected.  Because one of the 
> constituent elements of the thread's name is server.
> {code:java}
> Connection(ConnectionId remoteId, int serviceClass,
> Consumer removeMethod) {
> ..
> UserGroupInformation ticket = remoteId.getTicket();
> // try SASL if security is enabled or if the ugi contains tokens.
> // this causes a SIMPLE client with tokens to attempt SASL
> boolean trySasl = UserGroupInformation.isSecurityEnabled() ||
>   (ticket != null && !ticket.getTokens().isEmpty());
> this.authProtocol = trySasl ? AuthProtocol.SASL : AuthProtocol.NONE;
> this.setName("IPC Client (" + socketFactory.hashCode() +") connection to " +
> server.toString() +
> " from " + ((ticket==null)?"an unknown user":ticket.getUserName()));
> this.setDaemon(true);
> }{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16625) Backport HADOOP-14624 to branch-3.1

2019-10-03 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16625:
-
Fix Version/s: 3.1.4
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

Thanks!

> Backport HADOOP-14624 to branch-3.1
> ---
>
> Key: HADOOP-16625
> URL: https://issues.apache.org/jira/browse/HADOOP-16625
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.1.4
>
> Attachments: HADOOP-16625.branch-3.1.001.patch
>
>
> I am trying to bring commits from trunk/branch-3.2 to branch-3.1, but some of 
> them do not compile because of the commons-logging to slf4j migration. 
> One of the issue is GenericTestUtils.DelayAnswer do not accept slf4j logger 
> API.
> Backport HADOOP-14624 to branch-3.1 to make backport easier. It updates the 
> DelayAnswer signature, but it's in the test scope, so we're not really 
> breaking backward compat.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16544) update io.netty in branch-2

2019-10-03 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943775#comment-16943775
 ] 

Wei-Chiu Chuang commented on HADOOP-16544:
--

I just want to emphasize that netty 3.6.2Final is vulnerable to CVE-2014-0193 
(medium), CVE-2015-2156 (high)and CVE-2014-3488 (medium). I haven't verified if 
the change causes compat issue, but we really should move up.

> update io.netty in branch-2
> ---
>
> Key: HADOOP-16544
> URL: https://issues.apache.org/jira/browse/HADOOP-16544
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: release-blocker
> Fix For: 2.10.0
>
> Attachments: HADOOP-16544-branch-2.001.patch, 
> HADOOP-16544-branch-2.002.patch, HADOOP-16544-branch-2.003.patch, 
> HADOOP-16544-branch-2.004.patch
>
>
> branch-2 pulls in io.netty 3.6.2.Final which is more than 5 years old.
> The latest is 3.10.6Final. I know updating netty is sensitive but it deserves 
> some attention.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16588) Update commons-beanutils version to 1.9.4 in branch-2

2019-10-02 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943293#comment-16943293
 ] 

Wei-Chiu Chuang commented on HADOOP-16588:
--

+1 thank you.

> Update commons-beanutils version to 1.9.4 in branch-2
> -
>
> Key: HADOOP-16588
> URL: https://issues.apache.org/jira/browse/HADOOP-16588
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Critical
>  Labels: release-blocker
> Attachments: HADOOP-16588-branch-2.002.patch, 
> HADOOP-16588.branch-2.001.patch
>
>
> Similar to HADOOP-16542 but we need to do it differently.
> In branch-2, we pull in commons-beanutils through commons-configuration 1.6 
> --> commons-digester 1.8
> {noformat}
> [INFO] +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> {noformat}
> I have a patch to update version of the transitive dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16126) ipc.Client.stop() may sleep too long to wait for all connections

2019-10-02 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16126:
-
Fix Version/s: 3.2.2
   3.1.4

> ipc.Client.stop() may sleep too long to wait for all connections
> 
>
> Key: HADOOP-16126
> URL: https://issues.apache.org/jira/browse/HADOOP-16126
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc
>Reporter: Tsz-wo Sze
>Assignee: Tsz-wo Sze
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 2.9.3, 3.1.4, 3.2.2
>
> Attachments: c16126_20190219.patch, c16126_20190220.patch, 
> c16126_20190221.patch
>
>
> {code}
> //Client.java
>   public void stop() {
> ...
> // wait until all connections are closed
> while (!connections.isEmpty()) {
>   try {
> Thread.sleep(100);
>   } catch (InterruptedException e) {
>   }
> }
> ...
>   }
> {code}
> In the code above, the sleep time is 100ms.  We found that simply changing 
> the sleep time to 10ms could improve a Hive job running time by 10x.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16625) Backport HADOOP-14624 to branch-3.1

2019-10-02 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943180#comment-16943180
 ] 

Wei-Chiu Chuang commented on HADOOP-16625:
--

[~aajisaka] how do you think?

> Backport HADOOP-14624 to branch-3.1
> ---
>
> Key: HADOOP-16625
> URL: https://issues.apache.org/jira/browse/HADOOP-16625
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-16625.branch-3.1.001.patch
>
>
> I am trying to bring commits from trunk/branch-3.2 to branch-3.1, but some of 
> them do not compile because of the commons-logging to slf4j migration. 
> One of the issue is GenericTestUtils.DelayAnswer do not accept slf4j logger 
> API.
> Backport HADOOP-14624 to branch-3.1 to make backport easier. It updates the 
> DelayAnswer signature, but it's in the test scope, so we're not really 
> breaking backward compat.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16625) Backport HADOOP-14624 to branch-3.1

2019-10-02 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16625:
-
Status: Patch Available  (was: Open)

> Backport HADOOP-14624 to branch-3.1
> ---
>
> Key: HADOOP-16625
> URL: https://issues.apache.org/jira/browse/HADOOP-16625
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-16625.branch-3.1.001.patch
>
>
> I am trying to bring commits from trunk/branch-3.2 to branch-3.1, but some of 
> them do not compile because of the commons-logging to slf4j migration. 
> One of the issue is GenericTestUtils.DelayAnswer do not accept slf4j logger 
> API.
> Backport HADOOP-14624 to branch-3.1 to make backport easier. It updates the 
> DelayAnswer signature, but it's in the test scope, so we're not really 
> breaking backward compat.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16625) Backport HADOOP-14624 to branch-3.1

2019-10-02 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16625:
-
Attachment: HADOOP-16625.branch-3.1.001.patch

> Backport HADOOP-14624 to branch-3.1
> ---
>
> Key: HADOOP-16625
> URL: https://issues.apache.org/jira/browse/HADOOP-16625
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-16625.branch-3.1.001.patch
>
>
> I am trying to bring commits from trunk/branch-3.2 to branch-3.1, but some of 
> them do not compile because of the commons-logging to slf4j migration. 
> One of the issue is GenericTestUtils.DelayAnswer do not accept slf4j logger 
> API.
> Backport HADOOP-14624 to branch-3.1 to make backport easier. It updates the 
> DelayAnswer signature, but it's in the test scope, so we're not really 
> breaking backward compat.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Assigned] (HADOOP-16625) Backport HADOOP-14624 to branch-3.1

2019-10-02 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang reassigned HADOOP-16625:


Assignee: Wei-Chiu Chuang

> Backport HADOOP-14624 to branch-3.1
> ---
>
> Key: HADOOP-16625
> URL: https://issues.apache.org/jira/browse/HADOOP-16625
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-16625.branch-3.1.001.patch
>
>
> I am trying to bring commits from trunk/branch-3.2 to branch-3.1, but some of 
> them do not compile because of the commons-logging to slf4j migration. 
> One of the issue is GenericTestUtils.DelayAnswer do not accept slf4j logger 
> API.
> Backport HADOOP-14624 to branch-3.1 to make backport easier. It updates the 
> DelayAnswer signature, but it's in the test scope, so we're not really 
> breaking backward compat.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Created] (HADOOP-16625) Backport HADOOP-14624 to branch-3.1

2019-10-02 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HADOOP-16625:


 Summary: Backport HADOOP-14624 to branch-3.1
 Key: HADOOP-16625
 URL: https://issues.apache.org/jira/browse/HADOOP-16625
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


I am trying to bring commits from trunk/branch-3.2 to branch-3.1, but some of 
them do not compile because of the commons-logging to slf4j migration. 

One of the issue is GenericTestUtils.DelayAnswer do not accept slf4j logger API.

Backport HADOOP-14624 to branch-3.1 to make backport easier. It updates the 
DelayAnswer signature, but it's in the test scope, so we're not really breaking 
backward compat.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16161) NetworkTopology#getWeightUsingNetworkLocation return unexpected result

2019-10-01 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16161:
-
Fix Version/s: 3.2.2
   3.1.4

> NetworkTopology#getWeightUsingNetworkLocation return unexpected result
> --
>
> Key: HADOOP-16161
> URL: https://issues.apache.org/jira/browse/HADOOP-16161
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: net
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-16161.001.patch, HADOOP-16161.002.patch, 
> HADOOP-16161.003.patch, HADOOP-16161.004.patch, HADOOP-16161.005.patch, 
> HADOOP-16161.006.patch, HADOOP-16161.007.patch, HADOOP-16161.008.patch, 
> HADOOP-16161.009.patch
>
>
> Consider the following scenario:
> 1. there are 4 slaves and topology like:
> Rack: /IDC/RACK1
>hostname1
>hostname2
> Rack: /IDC/RACK2
>hostname3
>hostname4
> 2. Reader from hostname1, and calculate weight between reader and [hostname1, 
> hostname3, hostname4] by #getWeight, and their corresponding values are 
> [0,4,4]
> 3. Reader from client which is not in the topology, and in the same IDC but 
> in none rack of the topology, and calculate weight between reader and 
> [hostname1, hostname3, hostname4] by #getWeightUsingNetworkLocation, and 
> their corresponding values are [2,2,2]
> 4. Other different Reader can get the similar results.
> The weight result for case #3 is obviously not the expected value, the truth 
> is [4,4,4]. this issue may cause reader not really following arrange: local 
> -> local rack -> remote rack. 
> After dig the detailed implement, the root cause is 
> #getWeightUsingNetworkLocation only calculate distance between Racks rather 
> than hosts.
> I think we should add constant 2 to correct the weight of 
> #getWeightUsingNetworkLocation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16032) Distcp It should clear sub directory ACL before applying new ACL on it.

2019-10-01 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942382#comment-16942382
 ] 

Wei-Chiu Chuang commented on HADOOP-16032:
--

Pushed to branch-3.1. There is just a trivial conflict. Attached the patch for 
posterity.

> Distcp It should clear sub directory ACL before applying new ACL on it.
> ---
>
> Key: HADOOP-16032
> URL: https://issues.apache.org/jira/browse/HADOOP-16032
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools/distcp
>Affects Versions: 3.1.1
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.4
>
> Attachments: HADOOP-16032.000.patch, HADOOP-16032.001.patch, 
> HADOOP-16032.002.patch, HADOOP-16032.003.patch, HADOOP-16032.branch-3.1.patch
>
>
> Distcp preserve can't update the ACL info properly when source dir has access 
>  ACL and dest dir has default ACL. It will only modify the basic ACL part. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16032) Distcp It should clear sub directory ACL before applying new ACL on it.

2019-10-01 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16032:
-
Attachment: HADOOP-16032.branch-3.1.patch

> Distcp It should clear sub directory ACL before applying new ACL on it.
> ---
>
> Key: HADOOP-16032
> URL: https://issues.apache.org/jira/browse/HADOOP-16032
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools/distcp
>Affects Versions: 3.1.1
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HADOOP-16032.000.patch, HADOOP-16032.001.patch, 
> HADOOP-16032.002.patch, HADOOP-16032.003.patch, HADOOP-16032.branch-3.1.patch
>
>
> Distcp preserve can't update the ACL info properly when source dir has access 
>  ACL and dest dir has default ACL. It will only modify the basic ACL part. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16032) Distcp It should clear sub directory ACL before applying new ACL on it.

2019-10-01 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16032:
-
Fix Version/s: 3.1.4

> Distcp It should clear sub directory ACL before applying new ACL on it.
> ---
>
> Key: HADOOP-16032
> URL: https://issues.apache.org/jira/browse/HADOOP-16032
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools/distcp
>Affects Versions: 3.1.1
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.4
>
> Attachments: HADOOP-16032.000.patch, HADOOP-16032.001.patch, 
> HADOOP-16032.002.patch, HADOOP-16032.003.patch, HADOOP-16032.branch-3.1.patch
>
>
> Distcp preserve can't update the ACL info properly when source dir has access 
>  ACL and dest dir has default ACL. It will only modify the basic ACL part. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16140) hadoop fs expunge to add -immediate option to purge trash immediately

2019-10-01 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16140:
-
Fix Version/s: 3.1.4

> hadoop fs expunge to add -immediate option to purge trash immediately
> -
>
> Key: HADOOP-16140
> URL: https://issues.apache.org/jira/browse/HADOOP-16140
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 3.2.1, 3.1.4
>
> Attachments: HADOOP-14200.002.patch, HADOOP-14200.003.patch, 
> HADOOP-14200.004.patch, HADOOP-14200.005.patch, HADOOP-14200.006.patch, 
> HDFS-14200.001.patch
>
>
> I have always felt the HDFS trash is missing a simple way to empty the 
> current users trash immediately. We have "expunge" but in my experience 
> supporting clusters, end users find this confusing. When most end users run 
> expunge, they really want to empty their trash immediately and get confused 
> when expunge does not do this.
> This can result in users performing somewhat dangerous "skipTrash" operations 
> on the trash to free up space. The alternative, which most users will not 
> figure out on their own is:
> # Run the expunge command once - this will move the current folder to a 
> checkpoint and remove any old checkpoints older than the retention interval
> # Wait over 1 minute and then run expunge again, overriding fs.trash.interval 
> to 1 minute using the following command hadoop fs -Dfs.trash.interval=1 
> -expunge.
> With this Jira I am proposing to add a extra command, "hdfs dfs -emptyTrash" 
> that purges everything in the logged in users Trash directories immediately.
> How would the community feel about adding this new option? I will upload a 
> patch for comments.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Created] (HADOOP-16623) Expose total number of DT in JMX for KMS

2019-10-01 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HADOOP-16623:


 Summary: Expose total number of DT in JMX for KMS
 Key: HADOOP-16623
 URL: https://issues.apache.org/jira/browse/HADOOP-16623
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Wei-Chiu Chuang


Similar to HDFS-14449, we should expose total number of KMS delegation tokens 
in JMX.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15418) Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException

2019-10-01 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15418:
-
Fix Version/s: 3.2.2
   3.1.4

> Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of 
> iterator to avoid ConcurrentModificationException
> -
>
> Key: HADOOP-15418
> URL: https://issues.apache.org/jira/browse/HADOOP-15418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-15418.1.patch, HADOOP-15418.2.patch, 
> HADOOP-15418.3.patch
>
>
> The issue is similar to what was fixed in HADOOP-15411. Fixing this in 
> KMSAuthenticationFilter as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15418) Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException

2019-10-01 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942107#comment-16942107
 ] 

Wei-Chiu Chuang commented on HADOOP-15418:
--

Commit still applies in branch-3.1 and branch-3.2
I'll cherry pick it to these two branches.

> Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of 
> iterator to avoid ConcurrentModificationException
> -
>
> Key: HADOOP-15418
> URL: https://issues.apache.org/jira/browse/HADOOP-15418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15418.1.patch, HADOOP-15418.2.patch, 
> HADOOP-15418.3.patch
>
>
> The issue is similar to what was fixed in HADOOP-15411. Fixing this in 
> KMSAuthenticationFilter as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15909) KeyProvider class should implement Closeable

2019-10-01 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15909:
-
Fix Version/s: 3.2.2
   3.1.4

> KeyProvider class should implement Closeable
> 
>
> Key: HADOOP-15909
> URL: https://issues.apache.org/jira/browse/HADOOP-15909
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.3, 2.8.5
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-15909.001.patch
>
>
> KeyProviders and the extensions have close() methods. The classes should 
> implement Closeable that will allow try-with-resources to work and help catch 
> original exceptions instead of finally blocks and exception masking that 
> follows that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15909) KeyProvider class should implement Closeable

2019-10-01 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942098#comment-16942098
 ] 

Wei-Chiu Chuang commented on HADOOP-15909:
--

Commit still applies in branch-3.1 and branch-3.2.
Cherrypicking it into the two branches.

> KeyProvider class should implement Closeable
> 
>
> Key: HADOOP-15909
> URL: https://issues.apache.org/jira/browse/HADOOP-15909
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.3, 2.8.5
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15909.001.patch
>
>
> KeyProviders and the extensions have close() methods. The classes should 
> implement Closeable that will allow try-with-resources to work and help catch 
> original exceptions instead of finally blocks and exception masking that 
> follows that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16531) Log more detail for slow RPC

2019-09-30 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16531:
-
Fix Version/s: 3.2.2
   3.1.4

> Log more detail for slow RPC
> 
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
>   LOG.warn("Slow RPC : {} took {} {} to process from client {}",
>   methodName, processingTime, RpcMetrics.TIMEUNIT, call);
>   rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it 
> take to request lock, holding lock, or do other things)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16531) Log more detail for slow RPC

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941426#comment-16941426
 ] 

Wei-Chiu Chuang commented on HADOOP-16531:
--

Commit applies cleanly in branch-3.1 and branch-3.2. 
I'm going to cherry pick the commit into these two branches.

> Log more detail for slow RPC
> 
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
>   LOG.warn("Slow RPC : {} took {} {} to process from client {}",
>   methodName, processingTime, RpcMetrics.TIMEUNIT, call);
>   rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it 
> take to request lock, holding lock, or do other things)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

2019-09-30 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15865:
-
Fix Version/s: 3.2.2
   3.1.4

> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of 
> ConcurrentModificationException since we use iteration over Properties 
> object. 
> {code}
> private void overlay(Properties to, Properties from) {
>  for (Entry entry: from.entrySet()) {
>  to.put(entry.getKey(), entry.getValue());
>  }
>  }
> {code}
> Properties class is thread-safe but iterator is not. We should manually 
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by 
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: 
> Service ResourceManager failed in state STARTED; cause: 
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>  at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
>  at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
>  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
>  at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
>  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
>  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
>  at 
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
>  at 
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
>  at 
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
>  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
>  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
>  removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
>  Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
> 8032
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941425#comment-16941425
 ] 

Wei-Chiu Chuang commented on HADOOP-15865:
--

Patch applies cleanly in branch-3.1 and branch-3.2.
Cherry picking the commit into these two branches.

> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of 
> ConcurrentModificationException since we use iteration over Properties 
> object. 
> {code}
> private void overlay(Properties to, Properties from) {
>  for (Entry entry: from.entrySet()) {
>  to.put(entry.getKey(), entry.getValue());
>  }
>  }
> {code}
> Properties class is thread-safe but iterator is not. We should manually 
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by 
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: 
> Service ResourceManager failed in state STARTED; cause: 
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>  at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
>  at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
>  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
>  at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
>  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
>  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
>  at 
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
>  at 
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
>  at 
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
>  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
>  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
>  removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
>  Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
> 8032
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15014) KMS should log the IP address of the clients

2019-09-30 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15014:
-
Fix Version/s: 3.2.2
   3.1.4

> KMS should log the IP address of the clients
> 
>
> Key: HADOOP-15014
> URL: https://issues.apache.org/jira/browse/HADOOP-15014
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 2.8.1
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>Priority: Major
>  Labels: kms, log
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-15014.patch
>
>
> Currently KMSMDCFilter only captures http request url and method, but not the 
> remote address of the client.
>  Storing this information in a thread local variable would help external 
> authorizer plugins to do more thorough checks. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15014) KMS should log the IP address of the clients

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941424#comment-16941424
 ] 

Wei-Chiu Chuang commented on HADOOP-15014:
--

Commits apply cleanly in branch-3.2 and branch-3.1. I'm going to cherry pick 
them into these two branches.

> KMS should log the IP address of the clients
> 
>
> Key: HADOOP-15014
> URL: https://issues.apache.org/jira/browse/HADOOP-15014
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 2.8.1
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>Priority: Major
>  Labels: kms, log
> Fix For: 3.3.0
>
> Attachments: HADOOP-15014.patch
>
>
> Currently KMSMDCFilter only captures http request url and method, but not the 
> remote address of the client.
>  Storing this information in a thread local variable would help external 
> authorizer plugins to do more thorough checks. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock

2019-09-30 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16461:
-
Fix Version/s: 3.2.2
   3.1.4

> Regression: FileSystem cache lock parses XML within the lock
> 
>
> Key: HADOOP-16461
> URL: https://issues.apache.org/jira/browse/HADOOP-16461
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
>
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3388
> {code}
>   fs = createFileSystem(uri, conf);
>   synchronized (this) { // refetch the lock again
> FileSystem oldfs = map.get(key);
> if (oldfs != null) { // a file system is created while lock is 
> releasing
>   fs.close(); // close the new file system
>   return oldfs;  // return the old file system
> }
> // now insert the new file system into the map
> if (map.isEmpty()
> && !ShutdownHookManager.get().isShutdownInProgress()) {
>   ShutdownHookManager.get().addShutdownHook(clientFinalizer, 
> SHUTDOWN_HOOK_PRIORITY);
> }
> fs.key = key;
> map.put(key, fs);
> if (conf.getBoolean(
> FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
>   toAutoClose.add(key);
> }
> return fs;
>   }
> {code}
> The lock now has a ShutdownHook creation, which ends up doing 
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L205
> {code}
> HookEntry(Runnable hook, int priority) {
>   this(hook, priority,
>   getShutdownTimeout(new Configuration()),
>   TIME_UNIT_DEFAULT);
> }
> {code}
> which ends up doing a "new Configuration()" within the locked section.
> This indirectly hurts the cache hit scenarios as well, since if the lock on 
> this is held, then the other section cannot be entered either.
> https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/sort/impl/TezSpillRecord.java#L65
> {code}
> I/O Setup 0 State: BLOCKED CPU usage on sample: 6ms
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(URI, Configuration, 
> FileSystem$Cache$Key) FileSystem.java:3345
> org.apache.hadoop.fs.FileSystem$Cache.get(URI, Configuration) 
> FileSystem.java:3320
> org.apache.hadoop.fs.FileSystem.get(URI, Configuration) FileSystem.java:479
> org.apache.hadoop.fs.FileSystem.getLocal(Configuration) FileSystem.java:435
> {code}
> slowing down the RawLocalFileSystem when there are other threads creating 
> HDFS FileSystem objects at the same time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941416#comment-16941416
 ] 

Wei-Chiu Chuang commented on HADOOP-16461:
--

Commit applies to branch-3.2 and branch-3.1.
I'll cherry pick the commit into these two branches.

> Regression: FileSystem cache lock parses XML within the lock
> 
>
> Key: HADOOP-16461
> URL: https://issues.apache.org/jira/browse/HADOOP-16461
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 3.3.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Fix For: 3.3.0
>
>
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3388
> {code}
>   fs = createFileSystem(uri, conf);
>   synchronized (this) { // refetch the lock again
> FileSystem oldfs = map.get(key);
> if (oldfs != null) { // a file system is created while lock is 
> releasing
>   fs.close(); // close the new file system
>   return oldfs;  // return the old file system
> }
> // now insert the new file system into the map
> if (map.isEmpty()
> && !ShutdownHookManager.get().isShutdownInProgress()) {
>   ShutdownHookManager.get().addShutdownHook(clientFinalizer, 
> SHUTDOWN_HOOK_PRIORITY);
> }
> fs.key = key;
> map.put(key, fs);
> if (conf.getBoolean(
> FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
>   toAutoClose.add(key);
> }
> return fs;
>   }
> {code}
> The lock now has a ShutdownHook creation, which ends up doing 
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L205
> {code}
> HookEntry(Runnable hook, int priority) {
>   this(hook, priority,
>   getShutdownTimeout(new Configuration()),
>   TIME_UNIT_DEFAULT);
> }
> {code}
> which ends up doing a "new Configuration()" within the locked section.
> This indirectly hurts the cache hit scenarios as well, since if the lock on 
> this is held, then the other section cannot be entered either.
> https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/sort/impl/TezSpillRecord.java#L65
> {code}
> I/O Setup 0 State: BLOCKED CPU usage on sample: 6ms
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(URI, Configuration, 
> FileSystem$Cache$Key) FileSystem.java:3345
> org.apache.hadoop.fs.FileSystem$Cache.get(URI, Configuration) 
> FileSystem.java:3320
> org.apache.hadoop.fs.FileSystem.get(URI, Configuration) FileSystem.java:479
> org.apache.hadoop.fs.FileSystem.getLocal(Configuration) FileSystem.java:435
> {code}
> slowing down the RawLocalFileSystem when there are other threads creating 
> HDFS FileSystem objects at the same time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941357#comment-16941357
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

bq. Fortunately, the latest Curator 4.x versions are compatible with both 
ZooKeeper 3.4.x and 3.5.x.

So we don't have to update org.apache.zookeeper. Good.
That said, it would be a good idea to update zookeeper to 3.5.x. It gets rid of 
Netty 3, and we can gradually remove Netty 3 code in Hadoop: HADOOP-11219.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-11219) Upgrade to netty 4

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941332#comment-16941332
 ] 

Wei-Chiu Chuang commented on HADOOP-11219:
--

Netty 3 is EOL and the last Netty 3 was released on 6/2016, more than 3 years 
old.
We can't afford if Netty is found to have security vulnerability one day. Time 
to prioritize this work.

> Upgrade to netty 4
> --
>
> Key: HADOOP-11219
> URL: https://issues.apache.org/jira/browse/HADOOP-11219
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Major
>
> This is an umbrella jira to track the effort of upgrading to Netty 4.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941321#comment-16941321
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

The changes looks good to me. Let me know once you have additional test results.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16544) update io.netty in branch-2

2019-09-30 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941250#comment-16941250
 ] 

Wei-Chiu Chuang commented on HADOOP-16544:
--

We should.

> update io.netty in branch-2
> ---
>
> Key: HADOOP-16544
> URL: https://issues.apache.org/jira/browse/HADOOP-16544
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: release-blocker
> Attachments: HADOOP-16544-branch-2.001.patch, 
> HADOOP-16544-branch-2.002.patch, HADOOP-16544-branch-2.003.patch, 
> HADOOP-16544-branch-2.004.patch
>
>
> branch-2 pulls in io.netty 3.6.2.Final which is more than 5 years old.
> The latest is 3.10.6Final. I know updating netty is sensitive but it deserves 
> some attention.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15684) triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens.

2019-09-30 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15684:
-
Labels: multi-sbnn  (was: )

> triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException 
> happens. 
> 
>
> Key: HADOOP-15684
> URL: https://issues.apache.org/jira/browse/HADOOP-15684
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Rong Tang
>Assignee: Rong Tang
>Priority: Critical
>  Labels: multi-sbnn
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: 
> 0001-RollEditLog-try-next-NN-when-exception-happens.patch, 
> HADOOP-15684.000.patch, HADOOP-15684.001.patch, HADOOP-15684.002.patch, 
> HADOOP-15684.003.patch, HADOOP-15684.004.patch, HADOOP-15684.005.patch, 
> hadoop--rollingUpgrade-SourceMachine001.log
>
>
> When name node call triggerActiveLogRoll, and the cachedActiveProxy is a dead 
> name node, it will throws a ConnectTimeoutException, expected behavior is to 
> try next NN, but current logic doesn't do so, instead, it keeps trying the 
> dead, mistakenly take it as active.
>  
> 2018-08-17 10:02:12,001 WARN [Edit log tailer] 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a 
> roll of the active NN
> org.apache.hadoop.net.ConnectTimeoutException: Call From 
> SourceMachine001/SourceIP to001 TargetMachine001.ap.gbl:8020 failed on socket 
> timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 2 
> millis timeout 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:298)
>  
> C:\Users\rotang>ping TargetMachine001
> Pinging TargetMachine001[TargetIP001] with 32 bytes of data:
>  Request timed out.
>  Request timed out.
>  Request timed out.
>  Request timed out.
>  Attachment is a log file saying how it repeatedly retries a dead name node, 
> and a fix patch.
>  I replaced the actual machine name/ip as SourceMachine001/SourceIP001 and 
> TargetMachine001/TargetIP001.
>  
> How to Repro:
> In a good running NNs, take down the active NN (don't let it come back during 
> test), and then the stand by NNs will keep trying dead (old active) NN, 
> because it is the cached one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-09-26 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939081#comment-16939081
 ] 

Wei-Chiu Chuang commented on HADOOP-16579:
--

[~daryn] FYI I think you'd be interested in making connection to zookeeper more 
secure.

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-09-26 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16579:
-
Target Version/s: 3.3.0

> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version 
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
>  ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains 
> many new features (including SSL related improvements which can be very 
> important for production use; see [the release 
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier 
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
>  and [in Ozone we use Curator 
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, 
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 
> 3.5.x. (see [the relevant Curator 
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects 
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other 
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
>  - change Curator version in Hadoop to the latest stable 4.x version 
> (currently 4.2.0)
>  - also make sure we don't have multiple ZooKeeper versions in the classpath 
> to avoid runtime problems (it is 
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the 
> ZooKeeper which come with Curator, so that there will be only a single 
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in 
> Hadoop, we only want to make it possible for the community to be able to 
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the 
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper 
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with 
> both ZooKeeper 3.4 and 3.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16588) Update commons-beanutils version to 1.9.4 in branch-2

2019-09-20 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16588:
-
Attachment: HADOOP-16588.branch-2.001.patch

> Update commons-beanutils version to 1.9.4 in branch-2
> -
>
> Key: HADOOP-16588
> URL: https://issues.apache.org/jira/browse/HADOOP-16588
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Priority: Blocker
>  Labels: release-blocker
> Attachments: HADOOP-16588.branch-2.001.patch
>
>
> Similar to HADOOP-16542 but we need to do it differently.
> In branch-2, we pull in commons-beanutils through commons-configuration 1.6 
> --> commons-digester 1.8
> {noformat}
> [INFO] +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> {noformat}
> I have a patch to update version of the transitive dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16588) Update commons-beanutils version to 1.9.4 in branch-2

2019-09-20 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16588:
-
Priority: Critical  (was: Blocker)

> Update commons-beanutils version to 1.9.4 in branch-2
> -
>
> Key: HADOOP-16588
> URL: https://issues.apache.org/jira/browse/HADOOP-16588
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Priority: Critical
>  Labels: release-blocker
> Attachments: HADOOP-16588.branch-2.001.patch
>
>
> Similar to HADOOP-16542 but we need to do it differently.
> In branch-2, we pull in commons-beanutils through commons-configuration 1.6 
> --> commons-digester 1.8
> {noformat}
> [INFO] +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> {noformat}
> I have a patch to update version of the transitive dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Created] (HADOOP-16588) Update commons-beanutils version to 1.9.4 in branch-2

2019-09-20 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HADOOP-16588:


 Summary: Update commons-beanutils version to 1.9.4 in branch-2
 Key: HADOOP-16588
 URL: https://issues.apache.org/jira/browse/HADOOP-16588
 Project: Hadoop Common
  Issue Type: Task
Reporter: Wei-Chiu Chuang
 Attachments: HADOOP-16588.branch-2.001.patch

Similar to HADOOP-16542 but we need to do it differently.
In branch-2, we pull in commons-beanutils through commons-configuration 1.6 --> 
commons-digester 1.8

{noformat}
[INFO] +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
{noformat}

I have a patch to update version of the transitive dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16588) Update commons-beanutils version to 1.9.4 in branch-2

2019-09-20 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16588:
-
Labels: release-blocker  (was: )

> Update commons-beanutils version to 1.9.4 in branch-2
> -
>
> Key: HADOOP-16588
> URL: https://issues.apache.org/jira/browse/HADOOP-16588
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Priority: Blocker
>  Labels: release-blocker
> Attachments: HADOOP-16588.branch-2.001.patch
>
>
> Similar to HADOOP-16542 but we need to do it differently.
> In branch-2, we pull in commons-beanutils through commons-configuration 1.6 
> --> commons-digester 1.8
> {noformat}
> [INFO] +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> {noformat}
> I have a patch to update version of the transitive dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15804) upgrade to commons-compress 1.18

2019-09-13 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929680#comment-16929680
 ] 

Wei-Chiu Chuang commented on HADOOP-15804:
--

Cherrypicked the commit into branch2.

> upgrade to commons-compress 1.18
> 
>
> Key: HADOOP-15804
> URL: https://issues.apache.org/jira/browse/HADOOP-15804
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: PJ Fanning
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15804.01.patch
>
>
> [https://github.com/apache/commons-compress/blob/master/RELEASE-NOTES.txt]
> Some CVEs have been fixed in recent releases 
> ([https://commons.apache.org/proper/commons-compress/security-reports.html])
> [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.1.1] 
> depends on commons-compress 1.4.1



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19

2019-09-13 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16555:
-
Fix Version/s: 2.10.0

> Update commons-compress to 1.19
> ---
>
> Key: HADOOP-16555
> URL: https://issues.apache.org/jira/browse/HADOOP-16555
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: release-blocker
> Fix For: 2.10.0, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-16555.branch-3.2.patch
>
>
> We depend on commons-compress 1.18. The 1.19 release just went out. I think 
> we should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-15804) upgrade to commons-compress 1.18

2019-09-13 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15804:
-
Fix Version/s: 2.10.0

> upgrade to commons-compress 1.18
> 
>
> Key: HADOOP-15804
> URL: https://issues.apache.org/jira/browse/HADOOP-15804
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: PJ Fanning
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15804.01.patch
>
>
> [https://github.com/apache/commons-compress/blob/master/RELEASE-NOTES.txt]
> Some CVEs have been fixed in recent releases 
> ([https://commons.apache.org/proper/commons-compress/security-reports.html])
> [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.1.1] 
> depends on commons-compress 1.4.1



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Resolved] (HADOOP-16555) Update commons-compress to 1.19

2019-09-13 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HADOOP-16555.
--
Resolution: Fixed

There was a trivial conflict cherrypicking the commits into lower branches. 
Uploaded  [^HADOOP-16555.branch-3.2.patch]  for future reference. 

We may have to cherry pick this into branch-2.

> Update commons-compress to 1.19
> ---
>
> Key: HADOOP-16555
> URL: https://issues.apache.org/jira/browse/HADOOP-16555
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-16555.branch-3.2.patch
>
>
> We depend on commons-compress 1.18. The 1.19 release just went out. I think 
> we should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19

2019-09-13 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16555:
-
Fix Version/s: 3.1.3
   3.2.1

> Update commons-compress to 1.19
> ---
>
> Key: HADOOP-16555
> URL: https://issues.apache.org/jira/browse/HADOOP-16555
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-16555.branch-3.2.patch
>
>
> We depend on commons-compress 1.18. The 1.19 release just went out. I think 
> we should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19

2019-09-13 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16555:
-
Attachment: HADOOP-16555.branch-3.2.patch

> Update commons-compress to 1.19
> ---
>
> Key: HADOOP-16555
> URL: https://issues.apache.org/jira/browse/HADOOP-16555
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16555.branch-3.2.patch
>
>
> We depend on commons-compress 1.18. The 1.19 release just went out. I think 
> we should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19

2019-09-13 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16555:
-
Fix Version/s: 3.3.0

> Update commons-compress to 1.19
> ---
>
> Key: HADOOP-16555
> URL: https://issues.apache.org/jira/browse/HADOOP-16555
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0
>
>
> We depend on commons-compress 1.18. The 1.19 release just went out. I think 
> we should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Assigned] (HADOOP-16555) Update commons-compress to 1.19

2019-09-13 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang reassigned HADOOP-16555:


Assignee: YiSheng Lien

> Update commons-compress to 1.19
> ---
>
> Key: HADOOP-16555
> URL: https://issues.apache.org/jira/browse/HADOOP-16555
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
>  Labels: release-blocker
>
> We depend on commons-compress 1.18. The 1.19 release just went out. I think 
> we should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15977) RPC support for TLS

2019-09-11 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928046#comment-16928046
 ] 

Wei-Chiu Chuang commented on HADOOP-15977:
--

Thanks for the update, Daryn.
For the netty bugs, is there a public netty release that has the fix for the 
bugs?

> RPC support for TLS
> ---
>
> Key: HADOOP-15977
> URL: https://issues.apache.org/jira/browse/HADOOP-15977
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc, security
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Major
>
> Umbrella ticket to track adding TLS and mutual TLS support to RPC.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-09-10 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926732#comment-16926732
 ] 

Wei-Chiu Chuang commented on HADOOP-16152:
--

{noformat}
[WARNING] 
/testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java:[27,31]
 [deprecation] NCSARequestLog in org.eclipse.jetty.server has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java:[88,8]
 [deprecation] NCSARequestLog in org.eclipse.jetty.server has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java:[88,40]
 [deprecation] NCSARequestLog in org.eclipse.jetty.server has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java:[520,44]
 [deprecation] SslContextFactory() in SslContextFactory has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java:[521,23]
 [deprecation] setNeedClientAuth(boolean) in SslContextFactory has been 
deprecated
[WARNING] 
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpRequestLog.java:[21,31]
 [deprecation] NCSARequestLog in org.eclipse.jetty.server has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpRequestLog.java:[45,35]
 [deprecation] NCSARequestLog in org.eclipse.jetty.server has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestJettyHelper.java:[108,46]
 [deprecation] SslContextFactory() in SslContextFactory has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestJettyHelper.java:[109,25]
 [deprecation] setNeedClientAuth(boolean) in SslContextFactory has been 
deprecated
[WARNING] 
/testptch/hadoop/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java:[99,29]
 [deprecation] ConcurrentHashSet in org.eclipse.jetty.util has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java:[349,30]
 [deprecation] ConcurrentHashSet in org.eclipse.jetty.util has been deprecated
{noformat}
We need to deal with these.

I'm surprised by the shadedclient failure as it passed the build successfully 
for me.

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: https://issues.apache.org/jira/browse/HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16542) Update commons-beanutils version to 1.9.4

2019-09-10 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16542:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed to trunk.
I studied dependency tree, Hadoop branch-2 does not depend on commons-beanutils 
directly (there is an indirect dependency though) and the version pulled in is 
quite different.
{noformat}
[INFO] +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
{noformat}
Therefore, it may not be a trivial change for branch-2, and therefore I'd 
suggest to limit this commit to Hadoop 3.x

> Update commons-beanutils version to 1.9.4
> -
>
> Key: HADOOP-16542
> URL: https://issues.apache.org/jira/browse/HADOOP-16542
> Project: Hadoop Common
>  Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16542.001.patch, HADOOP-16542.002.patch, 
> HADOOP-16542.003.patch
>
>
> [http://mail-archives.apache.org/mod_mbox/www-announce/201908.mbox/%3cc628798f-315d-4428-8cb1-4ed1ecc95...@apache.org%3e]
>  {quote}
> CVE-2019-10086. Apache Commons Beanutils does not suppresses the class 
> property in PropertyUtilsBean
> by default.
> Severity: Medium
> Vendor: The Apache Software Foundation
> Versions Affected: commons-beanutils-1.9.3 and earlier
> Description: A special BeanIntrospector class was added in version 1.9.2.
> This can be used to stop attackers from using the class property of
> Java objects to get access to the classloader.
> However this protection was not enabled by default.
> PropertyUtilsBean (and consequently BeanUtilsBean) now disallows class
> level property access by default, thus protecting against
> CVE-2014-0114.
> Mitigation: 1.X users should migrate to 1.9.4.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16542) Update commons-beanutils version to 1.9.4

2019-09-10 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16542:
-
Affects Version/s: (was: 2.10.0)

> Update commons-beanutils version to 1.9.4
> -
>
> Key: HADOOP-16542
> URL: https://issues.apache.org/jira/browse/HADOOP-16542
> Project: Hadoop Common
>  Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16542.001.patch, HADOOP-16542.002.patch, 
> HADOOP-16542.003.patch
>
>
> [http://mail-archives.apache.org/mod_mbox/www-announce/201908.mbox/%3cc628798f-315d-4428-8cb1-4ed1ecc95...@apache.org%3e]
>  {quote}
> CVE-2019-10086. Apache Commons Beanutils does not suppresses the class 
> property in PropertyUtilsBean
> by default.
> Severity: Medium
> Vendor: The Apache Software Foundation
> Versions Affected: commons-beanutils-1.9.3 and earlier
> Description: A special BeanIntrospector class was added in version 1.9.2.
> This can be used to stop attackers from using the class property of
> Java objects to get access to the classloader.
> However this protection was not enabled by default.
> PropertyUtilsBean (and consequently BeanUtilsBean) now disallows class
> level property access by default, thus protecting against
> CVE-2014-0114.
> Mitigation: 1.X users should migrate to 1.9.4.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16542) Update commons-beanutils version to 1.9.4

2019-09-10 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16542:
-
Fix Version/s: 3.3.0

> Update commons-beanutils version to 1.9.4
> -
>
> Key: HADOOP-16542
> URL: https://issues.apache.org/jira/browse/HADOOP-16542
> Project: Hadoop Common
>  Issue Type: Task
>Affects Versions: 2.10.0, 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16542.001.patch, HADOOP-16542.002.patch, 
> HADOOP-16542.003.patch
>
>
> [http://mail-archives.apache.org/mod_mbox/www-announce/201908.mbox/%3cc628798f-315d-4428-8cb1-4ed1ecc95...@apache.org%3e]
>  {quote}
> CVE-2019-10086. Apache Commons Beanutils does not suppresses the class 
> property in PropertyUtilsBean
> by default.
> Severity: Medium
> Vendor: The Apache Software Foundation
> Versions Affected: commons-beanutils-1.9.3 and earlier
> Description: A special BeanIntrospector class was added in version 1.9.2.
> This can be used to stop attackers from using the class property of
> Java objects to get access to the classloader.
> However this protection was not enabled by default.
> PropertyUtilsBean (and consequently BeanUtilsBean) now disallows class
> level property access by default, thus protecting against
> CVE-2014-0114.
> Mitigation: 1.X users should migrate to 1.9.4.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16542) Update commons-beanutils version to 1.9.4

2019-09-10 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16542:
-
Summary: Update commons-beanutils version to 1.9.4  (was: Update 
commons-beanutils version)

> Update commons-beanutils version to 1.9.4
> -
>
> Key: HADOOP-16542
> URL: https://issues.apache.org/jira/browse/HADOOP-16542
> Project: Hadoop Common
>  Issue Type: Task
>Affects Versions: 2.10.0, 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Major
>  Labels: release-blocker
> Attachments: HADOOP-16542.001.patch, HADOOP-16542.002.patch, 
> HADOOP-16542.003.patch
>
>
> [http://mail-archives.apache.org/mod_mbox/www-announce/201908.mbox/%3cc628798f-315d-4428-8cb1-4ed1ecc95...@apache.org%3e]
>  {quote}
> CVE-2019-10086. Apache Commons Beanutils does not suppresses the class 
> property in PropertyUtilsBean
> by default.
> Severity: Medium
> Vendor: The Apache Software Foundation
> Versions Affected: commons-beanutils-1.9.3 and earlier
> Description: A special BeanIntrospector class was added in version 1.9.2.
> This can be used to stop attackers from using the class property of
> Java objects to get access to the classloader.
> However this protection was not enabled by default.
> PropertyUtilsBean (and consequently BeanUtilsBean) now disallows class
> level property access by default, thus protecting against
> CVE-2014-0114.
> Mitigation: 1.X users should migrate to 1.9.4.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16555) Update commons-compress to 1.19

2019-09-10 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16555:
-
Labels: release-blocker  (was: )

> Update commons-compress to 1.19
> ---
>
> Key: HADOOP-16555
> URL: https://issues.apache.org/jira/browse/HADOOP-16555
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Priority: Major
>  Labels: release-blocker
>
> We depend on commons-compress 1.18. The 1.19 release just went out. I think 
> we should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-09-10 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926540#comment-16926540
 ] 

Wei-Chiu Chuang commented on HADOOP-16152:
--

Thanks [~yumwang] for the work.

I think we should push this forward, especially since starting from 9.4.12 
Jetty supports TLS 1.3.

I think the biggest problem was shading. Spent today to get shading to work 
correctly. Attached is a patch that shades the necessary dependencies (between 
9.3.x and 9.4.x, jetty added a number of dependencies and they had to be shaded 
properly). We can use this patch as a starting point to test downstream 
applications. I also updated the Jetty version to the latest 9.4.20.

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: https://issues.apache.org/jira/browse/HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-09-10 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16152:
-
Attachment: HADOOP-16152.002.patch

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: https://issues.apache.org/jira/browse/HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Created] (HADOOP-16555) Update commons-compress to 1.19

2019-09-10 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HADOOP-16555:


 Summary: Update commons-compress to 1.19
 Key: HADOOP-16555
 URL: https://issues.apache.org/jira/browse/HADOOP-16555
 Project: Hadoop Common
  Issue Type: Task
Reporter: Wei-Chiu Chuang


We depend on commons-compress 1.18. The 1.19 release just went out. I think we 
should update it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16549) Remove Unsupported SSL/TLS Versions from Docs/Properties

2019-09-09 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16549:
-
Fix Version/s: 3.3.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

Thanks [~daisuke.kobayashi] and thanks [~aajisaka]!

> Remove Unsupported SSL/TLS Versions from Docs/Properties
> 
>
> Key: HADOOP-16549
> URL: https://issues.apache.org/jira/browse/HADOOP-16549
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, security
>Affects Versions: 3.3.0
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-16549.001.patch
>
>
> We should remove the following unsupported versions from docs and 
> core-default.xml appropriately.
> TLS v1.0
> TLS v1.1
> SSL v3
> SSLv2Hello
> ref: 
> https://www.eclipse.org/jetty/documentation/9.3.27.v20190418/configuring-ssl.html
> https://github.com/eclipse/jetty.project/issues/866
> [~aajisaka], I happened to find you left TLSv1.1 in 
> https://issues.apache.org/jira/browse/HADOOP-16000. Should we still keep it?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16549) Remove Unsupported SSL/TLS Versions from Docs/Properties

2019-09-09 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926266#comment-16926266
 ] 

Wei-Chiu Chuang commented on HADOOP-16549:
--

+1 will commit this shortly. Set target version 3.3.0.
For HADOOP-16152, it's a pretty tricky change actually. I'm hoping to get it 
into 3.3.0 but not sure.

> Remove Unsupported SSL/TLS Versions from Docs/Properties
> 
>
> Key: HADOOP-16549
> URL: https://issues.apache.org/jira/browse/HADOOP-16549
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, security
>Affects Versions: 3.3.0
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HADOOP-16549.001.patch
>
>
> We should remove the following unsupported versions from docs and 
> core-default.xml appropriately.
> TLS v1.0
> TLS v1.1
> SSL v3
> SSLv2Hello
> ref: 
> https://www.eclipse.org/jetty/documentation/9.3.27.v20190418/configuring-ssl.html
> https://github.com/eclipse/jetty.project/issues/866
> [~aajisaka], I happened to find you left TLSv1.1 in 
> https://issues.apache.org/jira/browse/HADOOP-16000. Should we still keep it?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Updated] (HADOOP-16549) Remove Unsupported SSL/TLS Versions from Docs/Properties

2019-09-09 Thread Wei-Chiu Chuang (Jira)


 [ 
https://issues.apache.org/jira/browse/HADOOP-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-16549:
-
Affects Version/s: 3.3.0

> Remove Unsupported SSL/TLS Versions from Docs/Properties
> 
>
> Key: HADOOP-16549
> URL: https://issues.apache.org/jira/browse/HADOOP-16549
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, security
>Affects Versions: 3.3.0
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HADOOP-16549.001.patch
>
>
> We should remove the following unsupported versions from docs and 
> core-default.xml appropriately.
> TLS v1.0
> TLS v1.1
> SSL v3
> SSLv2Hello
> ref: 
> https://www.eclipse.org/jetty/documentation/9.3.27.v20190418/configuring-ssl.html
> https://github.com/eclipse/jetty.project/issues/866
> [~aajisaka], I happened to find you left TLSv1.1 in 
> https://issues.apache.org/jira/browse/HADOOP-16000. Should we still keep it?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16549) Remove Unsupported SSL/TLS Versions from Docs/Properties

2019-09-09 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925609#comment-16925609
 ] 

Wei-Chiu Chuang commented on HADOOP-16549:
--

Also note HADOOP-16152 intends to upgrade to Jetty 9.4, and since 9.4.12 it 
supports TLS 1.3. So HADOOP-16152 should also add TLS1.3 into the enabled 
protocol.

> Remove Unsupported SSL/TLS Versions from Docs/Properties
> 
>
> Key: HADOOP-16549
> URL: https://issues.apache.org/jira/browse/HADOOP-16549
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, security
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HADOOP-16549.001.patch
>
>
> We should remove the following unsupported versions from docs and 
> core-default.xml appropriately.
> TLS v1.0
> TLS v1.1
> SSL v3
> SSLv2Hello
> ref: 
> https://www.eclipse.org/jetty/documentation/9.3.27.v20190418/configuring-ssl.html
> https://github.com/eclipse/jetty.project/issues/866
> [~aajisaka], I happened to find you left TLSv1.1 in 
> https://issues.apache.org/jira/browse/HADOOP-16000. Should we still keep it?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-16549) Remove Unsupported SSL/TLS Versions from Docs/Properties

2019-09-09 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925604#comment-16925604
 ] 

Wei-Chiu Chuang commented on HADOOP-16549:
--

LGTM thanks Daisuke

> Remove Unsupported SSL/TLS Versions from Docs/Properties
> 
>
> Key: HADOOP-16549
> URL: https://issues.apache.org/jira/browse/HADOOP-16549
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, security
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HADOOP-16549.001.patch
>
>
> We should remove the following unsupported versions from docs and 
> core-default.xml appropriately.
> TLS v1.0
> TLS v1.1
> SSL v3
> SSLv2Hello
> ref: 
> https://www.eclipse.org/jetty/documentation/9.3.27.v20190418/configuring-ssl.html
> https://github.com/eclipse/jetty.project/issues/866
> [~aajisaka], I happened to find you left TLSv1.1 in 
> https://issues.apache.org/jira/browse/HADOOP-16000. Should we still keep it?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



[jira] [Commented] (HADOOP-15169) "hadoop.ssl.enabled.protocols" should be considered in httpserver2

2019-09-08 Thread Wei-Chiu Chuang (Jira)


[ 
https://issues.apache.org/jira/browse/HADOOP-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925304#comment-16925304
 ] 

Wei-Chiu Chuang commented on HADOOP-15169:
--

[~aajisaka] I understand your concern. However, this is merely to achieve 
consistency with other Hadoop components. We've got customers with legacy tools 
that can only support SSLv2Hello, and they aren't able to use it after 
upgrading to Hadoop 3.

 

[~brahmareddy] thanks for the patch. have you tested it? Looking at Jetty's 
SslContextFactory implementation (SslContextFactory#selectProtocols()), after 
included protocols are added, it removes excluded protocols, which contains 

"SSL", "SSLv2", "SSLv2Hello", "SSLv3". I suspect we should reset excluded 
protocols before adding included protocols.

> "hadoop.ssl.enabled.protocols" should be considered in httpserver2
> --
>
> Key: HADOOP-15169
> URL: https://issues.apache.org/jira/browse/HADOOP-15169
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15169-branch-2.patch, HADOOP-15169.patch
>
>
> As of now *hadoop.ssl.enabled.protocols"* will not take effect for all the 
> http servers( only Datanodehttp server will use this config).



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org



  1   2   3   4   5   6   7   8   9   10   >