[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102296#comment-17102296 ] Andrea commented on HADOOP-16647: - This patch can be used hadoop2.6.0 > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Fix For: 3.3.0, 3.2.2, 2.10.1 > > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch, > HADOOP-16647-02.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087405#comment-17087405 ] Masatake Iwasaki commented on HADOOP-16647: --- cherry-picked this following HADOOP-14597, HADOOP-15062, HADOOP-16739. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Fix For: 3.3.0, 2.10.1 > > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch, > HADOOP-16647-02.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086475#comment-17086475 ] Luca Toscano commented on HADOOP-16647: --- [~iwasakims] thanks a lot for the commit, I am wondering if this change could also be backported to 2.10.x. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Fix For: 3.3.0 > > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch, > HADOOP-16647-02.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074960#comment-17074960 ] Hudson commented on HADOOP-16647: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18117 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18117/]) HADOOP-16647. Support OpenSSL 1.1.1 LTS. Contributed by Rakesh (iwasakims: rev 8f8be6b92a32982c2fd5be4647ebe1610a533dc3) * (edit) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch, > HADOOP-16647-02.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074635#comment-17074635
]
Hadoop QA commented on HADOOP-16647:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
51m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 10s{color} |
{color:red} root generated 4 new + 22 unchanged - 4 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
19s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HADOOP-16647 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998752/HADOOP-16647-02.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 29ff1c9ae0ff 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ff50ecd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| cc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16818/artifact/out/diff-compile-cc-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16818/testReport/ |
| Max. process+thread count | 1345 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16818/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch,
> HADOOP-16647-02.patch
>
>
> See Hadoop user mailing list
>
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074573#comment-17074573 ] Rakesh Radhakrishnan commented on HADOOP-16647: --- Attached new patch addressing [~iwasakims]'s comments. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch, > HADOOP-16647-02.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074290#comment-17074290
]
Masatake Iwasaki commented on HADOOP-16647:
---
[~rakeshr] I got no issue with the patch applied on Debian 9 with openssl
1.1.0l-1~deb9u1.
I'm +1 if formatting nits are addressed. {{#if}} and {{#endif}} should not have
leading whitespaces. The code inside the conditional should have same
indentation level with the outside.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch
>
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074212#comment-17074212 ] Wei-Chiu Chuang commented on HADOOP-16647: -- The cc warning doesn't look related. We can file a jira to remove that warning. Re: OpenSSLSecureRandom Hadoop 3.3.0 makes OpenSSLSecureRandom the default (HADOOP-16011) previously openssl crypto uses OsSecureRandom. Maybe that's we only see the runtime error now. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073903#comment-17073903 ] Rakesh Radhakrishnan commented on HADOOP-16647: --- Thanks a lot [~elukey] for the detailed reviews and suggestions. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073900#comment-17073900 ] Rakesh Radhakrishnan commented on HADOOP-16647: --- Thanks a lot [~iwasakims] for testing in different env and its really a useful feedback. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073783#comment-17073783 ] Masatake Iwasaki commented on HADOOP-16647: --- Thanks for working on this, [~rakeshr]. I could reproduce the error "java.lang.UnsatisfiedLinkError: CRYPTO_num_locks" on CentOS 8 with openssl-1.1.1-8.el8.x86_64. I got the error when I tried to put a file into Encryption Zone. There was no error on compilation and checknative. The HADOOP-16647-01.patch fixed the error. I also tried the HADOOP-16647-01.patch on CentOS 7 with openssl-1.0.2k-16.el7_6.1.x86_64. No issue on compilation and runtime. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073647#comment-17073647
]
Hadoop QA commented on HADOOP-16647:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
51m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 15s{color} |
{color:red} root generated 1 new + 25 unchanged - 1 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HADOOP-16647 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998587/HADOOP-16647-01.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 336834ee478d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4a3eb10 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| cc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16812/artifact/out/diff-compile-cc-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16812/testReport/ |
| Max. process+thread count | 1347 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16812/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch
>
>
> See Hadoop user mailing list
>
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073640#comment-17073640 ] Luca Toscano commented on HADOOP-16647: --- [~rakeshr] thanks for the clarification, now it looks really better and sound. I am not an expert in Openssl internals so the link provided looks reasonable, maybe given the importance of the patch we could reach our to openssl-user/dev@ to double check. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073570#comment-17073570
]
Rakesh Radhakrishnan commented on HADOOP-16647:
---
{quote}Rakesh, can you also check it can compile against openssl 1.0.2 as well
as 1.1.0/1.1.1?
{quote}
[~weichiu] Thanks for providing verification points, will try setup env and
verify in all these versions.
Hi [~elukey],
{quote}What versions of openssl/hadoop/os are you using to test? For example,
if you are on Debian, what version of libcrypto.so you have if you do ls -l
/usr/lib/x86_64-linux-gnu/libcrypto.so ?
{quote}
I am using libcrypto.so.1.1 and OpenSSL 1.1.1 11 Sep 2018
{quote}when I asked to run hadoop checknative as test, the error reported was
EVP_CIPHER_CTX_cleanup
{quote}
Yes, your observation is correct. HADOOP-14597 was not applied in my build and
caused EVP_CIPHER_CTX_cleanup error while exec {{hadoop checknative -a}}. Sorry
for the confusion due to my mistake. Now, I have upgraded the code and hits
{{java.lang.UnsatisfiedLinkError: CRYPTO_num_locks}} error during {{hdfs put
command}}.
Also, am attaching another patch covering only {{ifdefs}} in
OpensslSecureRandom.c
{quote}I am wondering if the locking code needs to happen also for 1.1.1, maybe
using something different than num_lock.
{quote}
Good point. I have referred following links and it says "OpenSSL 1.1.0+ "can be
safely used in multi-threaded applications provided that support for the
underlying OS threading API is built-in"
References:
[link-1|https://stackoverflow.com/questions/58224138/do-i-need-to-use-crypto-locking-functions-for-thread-safety-in-openssl-1-1-0]
and [link-2|https://curl.haxx.se/libcurl/c/threadsafe.html]
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch
>
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073379#comment-17073379 ] Luca Toscano commented on HADOOP-16647: --- [~weichiu] I'll explain my points, that were all for openssl 1.1.1 compatibility: * when I asked to run hadoop checknative as test, the error reported was EVP_CIPHER_CTX_cleanup, that in my experience comes from the fact that HADOOP-14597 is not applied, so I asked what was the testing environment to [~rakeshr]. * Judging from the error that you reported about CRYPTO_num_locks, it seems to me that the issue is in OpensslSecureRandom.c since it explicitly uses the function for locking purposes. Due to https://github.com/openssl/openssl/issues/1260, I see that the num_lock function has been moved to [crypto.h|https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/include/openssl/crypto.h#L212-L216] in openssl 1.1.1 and they are now a no-op, since the functionality is not supported anymore. My suggestion was to verify how to change OpensslSecureRandom.c's locking code to avoid using those functions, but something else. In theory openssl 1.1.0 and 1.0.2 should already be supported, if my understanding is correct we'd need to apply changes to the code only if openssl 1.1.1 is used. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073281#comment-17073281
]
Hadoop QA commented on HADOOP-16647:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
51m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 12s{color} |
{color:red} root generated 3 new + 23 unchanged - 3 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
22s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HADOOP-16647 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998459/HADOOP-16647-00.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 680f10b8ab5f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c613296 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| cc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16809/artifact/out/diff-compile-cc-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16809/testReport/ |
| Max. process+thread count | 2740 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16809/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch
>
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073237#comment-17073237 ] Wei-Chiu Chuang commented on HADOOP-16647: -- Worth noting that Apache Hadoop's precommit environment is on Ubuntu 16 (Xenial) so you'll not going to find out if the patch works with the precommit. https://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile#L21 > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073226#comment-17073226 ] Wei-Chiu Chuang commented on HADOOP-16647: -- I'm not sure I follow the conversation. Building Hadoop on openssl 1.1.0 was possible (HADOOP-14597). So I would suppose there are backward incompatible changes introduced in 1.1.1 not in 1.1.0? Or maybe Federa retained that symbol while the upstream openssl 1.1.0 removed that? The other thing is that openssl 1.0.2 is shipped with RHEL 7.4 and above, and Red Hat claims they are still supporting 1.0.2 till the end of support of RHEL 7. (RHEL 8 is on openssl 1.1.1 it looks like) https://access.redhat.com/solutions/2728111 Rakesh, can you also check it can compile against openssl 1.0.2 as well as 1.1.0/1.1.1? Not sure if openssl 1.1.0 was adopted widely. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > Attachments: HADOOP-16647-00.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072875#comment-17072875 ] Luca Toscano commented on HADOOP-16647: --- What I proposed is probably wrong, I just seen this on the openssl 1.1.1 code (crypto.h): /* * The old locking functions have been removed completely without compatibility * macros. This is because the old functions either could not properly report * errors, or the returned error values were not clearly documented. * Replacing the locking functions with no-ops would cause race condition * issues in the affected applications. It is far better for them to fail at * compile time. * On the other hand, the locking callbacks are no longer used. Consequently, * the callback management functions can be safely replaced with no-op macros. */ So I'd say that CRYPTO_num_locks and related should be removed (conditionally) in OpensslSecureRandom.c if openssl 1.1.1 is requested. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > Attachments: HADOOP-16647-00.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072858#comment-17072858 ] Luca Toscano commented on HADOOP-16647: --- I just did a quick test with Debian Buster on Docker, installing the hadoop package from BigTop 1.4 (containing the patch that I mentioned) and checknative -a returns to me openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so. I guess that checknative doesn't test OpensslSecureRandom.c, that is where the bug reported by [~weichiu] should come from. I don't have a test cluster with openssl 1.1.1 to test yarn on, but I'd try to simply add crypto.h to OpensslSecureRandom.c (or possibly only if openssl 1.1.1 is used as ifdef etc..) and see how it goes with a yarn map-reduce job. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > Attachments: HADOOP-16647-00.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072832#comment-17072832 ] Luca Toscano commented on HADOOP-16647: --- [~rakeshr] I think something may be wrong in your analysis, EVP_CIPHER_CTX_cleanup was removed IIRC on openssl 1.1.0 (not 1.1.1), and OpensslSecureRandom is already patched for this use case (I worked with BigTop devs in https://issues.apache.org/jira/browse/BIGTOP-3308 to backport a patch for hadoop 2.8.5 to allow hadoop check native -a to work on Debian 9, that ships openssl 1.1.0). I would expect some other result from hadoop checknative -a, but I'd have to test it. What versions of openssl/hadoop/os are you using to test? For example, if you are on Debian, what version of libcrypto.so you have if you do ls -l /usr/lib/x86_64-linux-gnu/libcrypto.so ? > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > Attachments: HADOOP-16647-00.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072765#comment-17072765
]
Rakesh Radhakrishnan commented on HADOOP-16647:
---
Attached initial draft patch to get early feedback, welcome comments, thanks!.
Basically the changes done to use openssl-1.1+ or openssl-1.0+ in Hadoop in
{{unix}} env.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Priority: Critical
> Attachments: HADOOP-16647-00.patch
>
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072757#comment-17072757
]
Rakesh Radhakrishnan commented on HADOOP-16647:
---
Thanks a lot [~elukey] for pointing out different cases. Sure will verify the
changes on MR or Spark job with encryption flow.
I saw that {{hadoop checknative -a}} failed with {{EVP_CIPHER_CTX_cleanup}}
function miss. I reviewed {{OpensslSecureRandom.c, OpensslCipher.c}} files and
saw that some more openssl functions to be considered for the compatibility.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Priority: Critical
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071010#comment-17071010 ] Luca Toscano commented on HADOOP-16647: --- Hi [~rakeshr], from what I can see [https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/include/openssl/crypto.h#L212-L216] the defines are now in crypto.h and they are yielding a no-op (more info [https://github.com/openssl/openssl/issues/1260]). OpensslSecureRandom.c seems not including crypto.h, so my guess is that this is the reason for the undefined reference error above. For testing I wouldn't rely only on hdfs put but also to the following (non-authoritative list, this is only a suggestion from my experience :)): * hdfs checknative -a (and check if openssl reports any error) * running a mapreduce job with encrypted shuffle or running a spark 2.4 job with AES encryption. Hope it helps! Luca > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070837#comment-17070837
]
Rakesh Radhakrishnan commented on HADOOP-16647:
---
How about adding a version check in
[OpensslSecureRandom.c|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c#L117]
file like below,
{code:java}
#if OPENSSL_VERSION_NUMBER < 0x1010L
// pre-1.1.0
LOAD_DYNAMIC_SYMBOL(dlsym_CRYPTO_num_locks, env, openssl,
"CRYPTO_num_locks");
LOAD_DYNAMIC_SYMBOL(dlsym_CRYPTO_set_locking_callback, \
env, openssl, "CRYPTO_set_locking_callback");
LOAD_DYNAMIC_SYMBOL(dlsym_CRYPTO_set_id_callback, env, \
openssl, "CRYPTO_set_id_callback");
#else
// post-1.1.0 version
#define dlsym_CRYPTO_num_locks() 1
#define dlsym_CRYPTO_set_locking_callback(a)
#define dlsym_CRYPTO_set_id_callback(a)
#endif{code}
I have tried an attempt in my test cluster using version check patch. Cluster
has {{OpenSSL 1.1.1 11 Sep 2018}} and {{Ubuntu 18.04.3}}
Here, I was able to run {{hdfs put}} command without the above mentioned
exception.
Welcome thoughts. Thanks!
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Priority: Critical
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064762#comment-17064762
]
Rakesh Radhakrishnan commented on HADOOP-16647:
---
*Reason for the failure:*
After checking the sources of openssl, I could see it's due to the changing of
openssl implementation in their latest version. It seems that OpenSSL breaks
its API with version 1.1.0+, {{CRYPTO_num_locks}} were in OpenSSL 1.0.2, but in
1.1.0+ they are macros so they no longer exist in the library.
[Openssl issue reference|https://github.com/openssl/openssl/issues/8726]
CRYPTO_num_locks function is available in older 1.0+ version [reference
here,|https://github.com/openssl/openssl/blob/OpenSSL_1_0_2-stable/crypto/cryptlib.c#L236]
but doesn’t exist in 1.1.0+ version, [reference
here|https://github.com/openssl/openssl/blob/OpenSSL_1_1_0-stable/crypto/cryptlib.c]
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Priority: Critical
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063556#comment-17063556
]
Wei-Chiu Chuang commented on HADOOP-16647:
--
So looks like HADOOP-16405 takes care of the openssl 1.1.1 in cloud connectors.
For the Hadoop services (YARN, HDFS), one of my colleague is taking a look at
it and found Hadoop does not currently run on openssl 1.1.1.
It generates error like the following:
{noformat}
20/03/20 05:20:07 ERROR random.OpensslSecureRandom: Failed to load Openssl
SecureRandom
java.lang.UnsatisfiedLinkError: CRYPTO_num_locks
at org.apache.hadoop.crypto.random.OpensslSecureRandom.initSR(Native
Method)
at
org.apache.hadoop.crypto.random.OpensslSecureRandom.(OpensslSecureRandom.java:57)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2598)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2563)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2659)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2685)
at
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.setConf(OpensslAesCtrCryptoCodec.java:59)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at org.apache.hadoop.crypto.CryptoCodec.getInstance(CryptoCodec.java:69)
at
org.apache.hadoop.hdfs.HdfsKMSUtil.getCryptoCodec(HdfsKMSUtil.java:110)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:961)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:947)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:538)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:532)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:546)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:473)
at
org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:195)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1133)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1113)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1002)
at
{noformat}
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Priority: Critical
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040978#comment-17040978 ] Luca Toscano commented on HADOOP-16647: --- Hi [~ste...@apache.org], thanks a lot for the info. Do you have a pointer to the azuredatalake/abfs by any chance? I am working with BigTop to see if the above change can be added to Hadoop 2.8.5/2.10, so any more info would be really appreciated. They already backported the changes made for HADOOP-14597, but as far as I can see on Debian 9 it is not enough even for openssl 1.1.0 (runtime issues when hadoop tries to use crypto libs provided by openssl). > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040966#comment-17040966 ] Steve Loughran commented on HADOOP-16647: - given its a breaker for 1.1.1, and compatible with the old releases, backport is fine if you are doing it though, there are some changes for azuredatalake and abfs which need to go in too related to moving off wildfly-1.0.4 and on to 1.0.7. Without that, you get to see NPEs > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040320#comment-17040320 ] Luca Toscano commented on HADOOP-16647: --- Should [https://github.com/apache/hadoop/commit/138c1ed5660f713d24bfebc44ea1846f76c00cb9] also be considered for backport to branch-2 (I suppose this means Hadoop 2.x)? I am currently working on BIGTOP-3308 to fix Debian 9's openssl 1.1.0 compatibility with BigTop 1.4 (Hadoop 2.8.5) and IIUC the aforementioned commit is essential to avoid functions deprecated in openssl 1.1.0 to be used. What do you think? > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Critical > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009626#comment-17009626 ] Gonzalo Gomez commented on HADOOP-16647: According to the OpenSSL blog, OpenSSL 1.0.2 reached its EOL ([https://www.openssl.org/policies/releasestrat.html]). Do you have any update on when will this new OpenSSL TLS version be supported? > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Major > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968265#comment-16968265 ] Gonzalo Gomez commented on HADOOP-16647: I see you are adding this issue to the 3.3.0 release. Do you have an estimation of when it will be released? > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Priority: Major > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
