[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2020-04-20 Thread Masatake Iwasaki (Jira) 


 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HADOOP-14597:
--
Fix Version/s: 2.10.1

> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>Priority: Major
> Fix For: 3.0.0-beta1, 2.10.1
>
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch, 
> HADOOP-14597.02.patch, HADOOP-14597.03.patch, HADOOP-14597.04.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2017-07-24 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-14597:
--
   Resolution: Fixed
Fix Version/s: 3.0.0-beta1
   Status: Resolved  (was: Patch Available)

> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch, 
> HADOOP-14597.02.patch, HADOOP-14597.03.patch, HADOOP-14597.04.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2017-06-29 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-14597:
--
Attachment: HADOOP-14597.04.patch

Aah! Good point. Thank you Allen! Done.

Are you aware of any tests which I can run to check functionality? I don't see 
anything in the test directory for native encryption.

Good point about {{EVP_CIPHER_CTX_encrypting}}. The function doesn't exist in 
OpenSSL-1.0.2 . I've put an ugly ifdef for that too now. I'm not sure what else 
I could do now, except dilute the semantics (really its just a one byte 
difference on checking buffer bounds depending on whether its encrypting or 
decrypting). Or we could set minimum version of OpenSSL to 1.1.0 which I'm sure 
will make a lot of people unhappy.

[~hitliuyi] Could you also please review this?

> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch, 
> HADOOP-14597.02.patch, HADOOP-14597.03.patch, HADOOP-14597.04.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2017-06-28 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-14597:
--
Attachment: HADOOP-14597.03.patch

Thanks Allen! I tested this one with OpenSSL-1.0.2. I had to pull in 
MAPREDUCE-6536 too. 

FWIW I built OpenSSL-1.0.2 from source using {code} ./config 
--prefix=/some/directory; make; make install {code}
To let common build using {{-Dopenssl.prefix=/some/directory}}, I had to 
{code}diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt 
b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
index 10b0f23ddf..01aadd546a 100644
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -176,6 +176,7 @@ if(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
 SET(OPENSSL_NAME "eay32")
 endif()
 message("CUSTOM_OPENSSL_PREFIX = ${CUSTOM_OPENSSL_PREFIX}")
+SET(CMAKE_FIND_LIBRARY_SUFFIXES ".so" ".a")
 find_library(OPENSSL_LIBRARY
 NAMES ${OPENSSL_NAME}
 PATHS ${CUSTOM_OPENSSL_PREFIX} ${CUSTOM_OPENSSL_PREFIX}/lib
{code}
Seems like we are playing games to be better at detecting OpenSSL than Cmake 
:-) . I know I'm way off on a tangent now ;-) Hahaha


> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch, 
> HADOOP-14597.02.patch, HADOOP-14597.03.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2017-06-27 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-14597:
--
Attachment: HADOOP-14597.02.patch

HADOOP-14597.00.patch and HADOOP-14597.01.patch are bad. They are passing a 
[{{EVP_CIPHER_CTX}}|https://github.com/openssl/openssl/blob/master/crypto/evp/evp_locl.h#L24]
 to 
[{{EVP_CIPHER_flags}}|https://github.com/openssl/openssl/blob/master/crypto/evp/evp_lib.c#L196]
 which actually expects 
[{{EVP_CIPHER}}|https://github.com/openssl/openssl/blob/master/crypto/include/internal/evp_int.h#L115]
 . These are two different structs (none inheriting the other).
I am not using the proper method and I'm guessing the openssl devs intended for 
us to use it this way. 

Can someone please review and commit?

> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch, 
> HADOOP-14597.02.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2017-06-27 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-14597:
--
Target Version/s: 3.0.0-alpha4
  Status: Patch Available  (was: Open)

> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch, 
> HADOOP-14597.02.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2017-06-27 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-14597:
--
Attachment: HADOOP-14597.01.patch

This patch lets hadoop-pipes compile too

> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
> Attachments: HADOOP-14597.00.patch, HADOOP-14597.01.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

2017-06-27 Thread Ravi Prakash (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-14597:
--
Attachment: HADOOP-14597.00.patch

I'm not sure this is the right patch, but it lets the compilation continue. I'm 
using functions defined here 
https://github.com/openssl/openssl/blob/master/crypto/evp/evp_lib.c#L196. I 
don't know what the intent of making the internals opaque is, if I can get to 
the same flags using this method. This leads me to believe this patch is 
probably not what the openssl devs intended. Anyone know anything about this?

> Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been 
> made opaque
> 
>
> Key: HADOOP-14597
> URL: https://issues.apache.org/jira/browse/HADOOP-14597
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha4
> Environment: openssl-1.1.0
>Reporter: Ravi Prakash
> Attachments: HADOOP-14597.00.patch
>
>
> Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
> with this error
> {code}[WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
>  In function ‘check_update_max_output_len’:
> [WARNING] 
> /home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
>  error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
> evp_cipher_ctx_st}’
> [WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
> [WARNING]   ^~
> {code}
> https://github.com/openssl/openssl/issues/962 mattcaswell says
> {quote}
> One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
> version is that many types have been made opaque, i.e. applications are no 
> longer allowed to look inside the internals of the structures
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org