[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9238 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 2:

I manually triggered a gerrit-verify-dryrun-external. And the UT passed.
Would you please re-trigger Jenkins.
Thanks a lot
 > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1910/


--
To view, visit http://gerrit.cloudera.org:8080/9238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia188a0c5b74e4a22fb30f8c12f65e0469eb75f6b
Gerrit-Change-Number: 9238
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Fri, 09 Feb 2018 13:43:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9238 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 1:

@Tim, I've fixed the compiling error on centos 6.4. Would you please help 
review.
 > Uploaded patch set 1.


--
To view, visit http://gerrit.cloudera.org:8080/9238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia188a0c5b74e4a22fb30f8c12f65e0469eb75f6b
Gerrit-Change-Number: 9238
Gerrit-PatchSet: 1
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Wed, 07 Feb 2018 17:04:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9238


Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: Ia188a0c5b74e4a22fb30f8c12f65e0469eb75f6b
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 232 insertions(+), 78 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/9238/1
--
To view, visit http://gerrit.cloudera.org:8080/9238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia188a0c5b74e4a22fb30f8c12f65e0469eb75f6b
Gerrit-Change-Number: 9238
Gerrit-PatchSet: 1
Gerrit-Owner: Xianda Ke 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9032 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 14:

Hi Sailesh,  I've fixed the clang-tidy issue. And the gerrit-dryrun test 
passed.  Please help commit. Thanks a lot.
https://jenkins.impala.io/job/gerrit-verify-dryrun-external/73/console


--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 14
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Sat, 03 Feb 2018 16:39:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#14).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 227 insertions(+), 73 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/14
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 14
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#13).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 230 insertions(+), 73 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/13
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 13
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9032 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 11:

fix merge conflict & slight change (move a #include statement to .cc file)
 > Uploaded patch set 11.


--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 11
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 31 Jan 2018 08:10:51 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#11).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 230 insertions(+), 73 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/11
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 11
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9032 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 10:

Thank you for the review. I'll fix the merge conflict and update the patch.
 > (1 comment)


--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 10
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 31 Jan 2018 06:56:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9032 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 10:

(1 comment)

> (1 comment)

Yes. It could be false if we are running on a system without GCM support.

 > (1 comment)

http://gerrit.cloudera.org:8080/#/c/9032/9/be/src/util/openssl-util-test.cc
File be/src/util/openssl-util-test.cc:

http://gerrit.cloudera.org:8080/#/c/9032/9/be/src/util/openssl-util-test.cc@149
PS9, Line 149: // Check if GCM mode
> Any update on this comment?
Thank Sailesh for the comments. Even it has been set as GCM mode, it may fall 
back to other modes. So no DCHECK needed. I just add a line of comment in Patch 
Set 10. Thanks a lot



--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 10
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 31 Jan 2018 06:38:13 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#10).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 231 insertions(+), 73 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/10
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 10
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9032 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 9:

Thank Tim a lot for the review.  @Sailesh, would you please help review 
IsModeSupported() in openssl-util.cc.


--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 9
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 24 Jan 2018 03:21:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#9).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 232 insertions(+), 78 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/9
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 9
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#8).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 232 insertions(+), 78 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/8
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 8
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9032 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 6:

(4 comments)

> Patch Set 6: Published edit on patch set 5.

Thank Zoltan for your comments. The duplicated logic are removed.
I prefer the "falling back" option. if we provide conf option to the user. If 
user want to use GCM mode but still there are some 'old' nodes do NOT support 
pclmulqdq instruction in the cluster. The "falling back" option doesn't block 
the user. Zoltan, any comments?

http://gerrit.cloudera.org:8080/#/c/9032/4/be/src/util/cpu-info.h
File be/src/util/cpu-info.h:

http://gerrit.cloudera.org:8080/#/c/9032/4/be/src/util/cpu-info.h@37
PS4, Line 37:= (1 << 1);
:   static const int64_t SSE4_1= (1 << 2);
:   static const int64_t SSE4_2= (1 << 3);
:   static const int64_t POPCNT= (1 << 4);
:   static const int64_t AVX   = (1 << 5);
:   static const int64_t AVX2  = (1 << 6);
:   static const int64_t PCLMULQDQ =
> please align the assignment operators
fixed


http://gerrit.cloudera.org:8080/#/c/9032/4/be/src/util/openssl-util-test.cc
File be/src/util/openssl-util-test.cc:

http://gerrit.cloudera.org:8080/#/c/9032/4/be/src/util/openssl-util-test.cc@118
PS4, Line 118:
 : /// Test that encryption and decryption work in-place.
 : TEST_F(OpenSSLUtilTest, EncryptInPlace) {
 :   const int buffer_size = 1024 * 1024;
 :   TestEncryptionDecryption(buffer_size);
 : }
 :
 : /// Test that encryption works with buffer lengths that don't 
fit in a 32-bit integer.
 : TEST_F(OpenSSLUtilTest, EncryptInPlaceHugeBuffer) {
 :   const int64_t buffer_size = 3 * 1024L * 1024L * 1024L;
 :   TestEncryptionDecryption(buffer_size);
 : }
 :
 : /// Test that encryption works with arbitrary-length buffer
 : TEST_F(OpenSSLUtilTest, EncryptArbitraryLength) {
 :   std::uniform_int_distribution dis(0, 1024 * 1024);
 :   const int buffer_size = dis(rng_);
 :   TestEncryptionDecryption(buffer_size);
 : }
 :
 : /// Test integrity in GCM mode
 : TEST_F(OpenSSLUtilTest, GcmIntegrity) {
 :   const int buffer_size = 1024 * 1024;
 :   vector buffer(buffer_size);
 :
 :   EncryptionKey key;
 :   key.InitializeRandom();
 :   key.SetCipherMode(AES_256_GCM);
 :
 :   // If GCM mode is supported at runtime
 :   if (key.IsGcmMode()) {
 : GenerateRandomData(buffer.data(), buffer_size);
 : ASSERT_OK(key.Encrypt(buffer.data(), buffer_size, 
buffer.data()));
 :
 : // tamper the data
 : ++buffer[0];
 : Status s = key.Decrypt(buffer.data(), buffer_size, 
buffer.data());
 : EXPECT_STR_CONTAINS(s.GetDetail(), "EVP_DecryptFinal");
 :   }
 : }
 :
 : /// Test basic integrity hash functionality.
 : TEST_F(OpenSSLUtilTest, IntegrityHash) {
 :   const int buffer_size = 1024 * 1024;
 :   vector buf1(buffer_size);
 :   vector buf1_copy(buffer_size);
 :   vector buf2(buffer_size);
 :
> These two tests are pretty similar. You could refactor the identical parts
three functions are refactored


http://gerrit.cloudera.org:8080/#/c/9032/4/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:

http://gerrit.cloudera.org:8080/#/c/9032/4/be/src/util/openssl-util.cc@162
PS4, Line 162: // use weak symbol to avoid compiling error on OpenSSL 1.0.0 
environment
> The weak symbol test is now in the constructor of EncryptionKey, I think th
refactored


http://gerrit.cloudera.org:8080/#/c/9032/4/be/src/util/openssl-util.cc@173
PS4, Line 173: LOG(WARNING) << "This mode is not supported, fall back to 
the default mode.";
 : mode_ = GetSupportedDefaultMode();
 :   }
 : }
 :
 : bool EncryptionKey::IsModeSupported(AES_CIPHER_MODE m) const {
 :   switch (m) {
 : case AES_256_GCM:
 :   return (CpuInfo::IsSupported(CpuInfo::PCLMULQDQ)
 :   && SSLeay() >= OPENSSL_VERSION_1_0_1D && 
EVP_aes_256_gcm);
 :
 : case AES_256_CTR:
 :
> This part of the code is very similar to the constructor.
good suggestion. Thanks a lot! duplicated logic is refactored. "falling chain" 
is same now.
I prefer the "falling back" option. if we provide conf option to the user. If 
user want to use GCM mode but still 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#6).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 194 insertions(+), 66 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/6
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9032

to look at the new patch set (#5).

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 194 insertions(+), 66 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/5
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 5
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Reviewer: Zoltan Borok-Nagy 

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9032 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 4:

Hi folks, Would you please help review when you are available?
@Tim, Would you please help run the test on all the platforms(old centos...)


--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Thu, 18 Jan 2018 00:43:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Xianda Ke (Code Review)]

Xianda Ke has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9032


Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..

IMPALA-6219: Use AES-GCM for spill-to-disk encryption

AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it
requires an instruction that Impala can currently run without (CLMUL).
In order to be fast, we dispatch to GCM mode at run-time based on the
CPU and OpenSSL version.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test
and buffered-tuple-stream-test.
add two cases GcmIntegrity & EncryptoArbitraryLength for
openssl-util-test

Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/cpu-info.cc
M be/src/util/cpu-info.h
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
6 files changed, 176 insertions(+), 37 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/4
--
To view, visit http://gerrit.cloudera.org:8080/9032
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24
Gerrit-Change-Number: 9032
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke 

[Impala-ASF-CR] IMPALA-6128: Add support for AES-CTR encryption when spilling to disk CFB mode is a stream cipher and is secure when used with a different nonce/IV for every message. However it can be

[Xianda Ke (Code Review)]

Hello Sailesh Mukil, Tim Armstrong, Bikramjeet Vig,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8861

to look at the new patch set (#4).

Change subject: IMPALA-6128: Add support for AES-CTR encryption when spilling 
to disk CFB mode is a stream cipher and is secure when used with a different 
nonce/IV for every message. However it can be a performance bottleneck. CTR 
mode is also stream cipher and is secure
..

IMPALA-6128: Add support for AES-CTR encryption when spilling to disk
CFB mode is a stream cipher and is secure when used with a different nonce/IV
for every message. However it can be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster than CFB mode in
OpenSSL. AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256.

CTR mode is used if OpenSSL version>=1.0.1 at runtime, otherwise
fall back to using CFB mode.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and
buffered-tuple-stream-test
The ut case openssl-util-test.EncryptInPlace tests encryption in both modes.

Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
4 files changed, 94 insertions(+), 44 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/8861/4
--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow CFB mode is stream cipher and is secure when use different nonce/IV for every message. However it would be a performance

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8861 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow 
CFB mode is stream cipher and is secure when use different nonce/IV for every 
message. However it would be a performance bottleneck. CTR mode is also stream 
cipher and is secure, 4~6x faster
..


Patch Set 2:

Thanks for you comments.  Merry Xmax!


--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 24 Dec 2017 14:45:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow CFB mode is stream cipher and is secure when use different nonce/IV for every message. However it would be a performance

[Xianda Ke (Code Review)]

Xianda Ke has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/8861


Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow 
CFB mode is stream cipher and is secure when use different nonce/IV for every 
message. However it would be a performance bottleneck. CTR mode is also stream 
cipher and is secure, 4~6x faster
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL. AES-CTR+SHA256 is about 40~70% faster
than AES-CFB+SHA256.

CTR mode is used if OpenSSL version>=1.0.1 at runtime, otherwise
fall back to CFB mode.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
4 files changed, 70 insertions(+), 24 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/8861/2
--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 

[Impala-ASF-CR] IMPALA-6220: Revert "IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow"

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8597 )

Change subject: IMPALA-6220: Revert "IMPALA-6128: Spill-to-disk 
Encryption(AES-CFB + SHA256) is slow"
..


Patch Set 1: Code-Review+1

> Assignee added: Sailesh Mukil 

it is ok to revert since it blocks compiling now.
I'll investigate it to find out a solution for CTR mode.


--
To view, visit http://gerrit.cloudera.org:8080/8597
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id31d5fcfec5c6d777d4acee5c1be2d4fc4605efb
Gerrit-Change-Number: 8597
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Mon, 20 Nov 2017 09:23:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 6:

(1 comment)

Thank Sailesh for review. fixed now.
I will use clang-format-diff.py for formatting next time.
> Uploaded patch set 6.

http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:

http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc@107
PS2, Line 107:   const EVP_CIPHER* evpCipher = IsCtrSupported() ? 
EVP_aes_256_ctr() : EVP_aes_256_cfb();
> long line
thank you for review. these two issues are fixed now



--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 19 Nov 2017 05:03:04 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

[Xianda Ke (Code Review)]

Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#6).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/6
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

[Xianda Ke (Code Review)]

Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#4).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 12 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/4
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 2:

> Thanks for doing this. It seems reasonable to me to switch to CTR.
 >
 > One concern I have is were you able to test this on a system with
 > OpenSSL 1.0.0? Their documentation isn't clear, so I'm not sure if
 > this API is supported in that version.
 > The best I could see was this line of code that was under an #if 0:
 > https://github.com/openssl/openssl/blob/OpenSSL_1_0_0-stable/crypto/evp/evp.h#L782-L783
 >
 > It would be great if you could verify that.

Thank Sailesh for your comments. aes_256_ctr was supported since openssl 1.0.1. 
Luckily, Impala add openssl to toolchain and upgrade to version 1.0.1p (see 
IMPALA-5714)


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Thu, 16 Nov 2017 14:55:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

[Xianda Ke (Code Review)]

Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#3).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 11 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/3
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 3
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong