[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262055#comment-14262055 ] Lefty Leverenz commented on HIVE-9167: -- Does this need any documentation? (For example, CRYPTO in HiveCommand.java and an explanation of how to create keys zones in .q files.) Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Labels: Kanban Fix For: encryption-branch Attachments: HIVE-9167.4.patch The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262234#comment-14262234 ] Ferdinand Xu commented on HIVE-9167: Hi [~leftylev], I don't think crypto command needs documentations since it is visible for test only. Thank you! Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Labels: Kanban Fix For: encryption-branch Attachments: HIVE-9167.4.patch The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262490#comment-14262490 ] Lefty Leverenz commented on HIVE-9167: -- Thanks [~xu], and Happy New Year! Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Labels: Kanban Fix For: encryption-branch Attachments: HIVE-9167.4.patch The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261279#comment-14261279 ] Brock Noland commented on HIVE-9167: Hi, Thank you Sergio! I am going to go ahead and commit this since you will be out after today. We can address and remaining issues as follow-on jiras. Thank you! Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Labels: Hive-Scrum Attachments: HIVE-9167.4.patch The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[
https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261325#comment-14261325
]
Hive QA commented on HIVE-9167:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689535/HIVE-9167.4.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2221/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2221/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2221/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2221/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'beeline/src/java/org/apache/hive/beeline/Commands.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target itests/target itests/hcatalog-unit/target
itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target
itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target
itests/util/target hcatalog/target hcatalog/core/target
hcatalog/streaming/target hcatalog/server-extensions/target
hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target accumulo-handler/target hwi/target
common/target common/src/gen contrib/target service/target serde/target
beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
U
ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java
Uql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q
Dql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out
A
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.7.out
A
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.8.out
Uql/src/test/results/clientnegative/unset_table_property.q.out
Dql/src/test/results/clientpositive/list_bucket_dml_10.q.out
Uql/src/test/results/clientpositive/stats_list_bucket.q.java1.8.out
Uql/src/test/results/clientpositive/multiMapJoin2.q.out
Uql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.8.out
Uql/src/test/results/clientpositive/auto_join_without_localtask.q.out
Aql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.7.out
Aql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.8.out
Uql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
Uql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
U
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
U
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java
U
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1648561.
Updated to revision 1648561.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260374#comment-14260374 ] Sergio Peña commented on HIVE-9167: --- Hi [~brocknoland]. I am still doing some changes on that RB patch. I'll let you know when it is ready. Thanks [~Ferd] for the comments. I working on those and other changes. Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259786#comment-14259786 ] Brock Noland commented on HIVE-9167: Hi [~spena], I see a RB item but I cannot remember if this patch was ready? Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259796#comment-14259796 ] Ferdinand Xu commented on HIVE-9167: Hi [~spena], I have a few comments left on your review board entry. By adding this crypto_helper command, how can we create two keys in different length since they are specified in the configuration? Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[
https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258960#comment-14258960
]
Dong Chen commented on HIVE-9167:
-
The approach looks good!
About not exposing the command to the end user, maybe we can leave the value of
{{hive.security.command.whitelist}} in HiveConf as original, and adding the
command into whitelist in conf when encryption test initialization. How does
this sound? It is a simple way, although it does not really hide the cmd from
user.
Enhance encryption testing framework to allow create keys zones inside .q
files
-
Key: HIVE-9167
URL: https://issues.apache.org/jira/browse/HIVE-9167
Project: Hive
Issue Type: Sub-task
Reporter: Sergio Peña
Assignee: Sergio Peña
The current implementation of the encryption testing framework on HIVE-8900
initializes a couple of encrypted databases to be used on .q test files. This
is useful in order to make tests small, but it does not test all details
found on the encryption implementation, such as: encrypted tables with
different encryption strength in the same database.
We need to allow this kind of encryption as it is how it will be used in the
real world where a database will have a few encrypted tables (not all the DB).
Also, we need to make this encryption framework flexible so that we can
create/delete keys zones on demand when running the .q files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14253514#comment-14253514 ] Sergio Peña commented on HIVE-9167: --- Hi [~Ferd], What I am trying to do with the crypto commands is that we can create encryption zones for specific tables during our tests. Users will have tables with different encryption zones, and I'd like to test different queries that work with these tables. The encryption support in Hive does a Copy of data when a query uses two tables with different encryption zones, and a Move when they're in the same encryption zone (an encrypted DB is just an encryption zone, so all queries will use Move). Also, I think this looks more readable when another developer reads the .q files. Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14253679#comment-14253679 ] Brock Noland commented on HIVE-9167: bq. do we real need add the crypto command for the beeline? Sergio mentioned this to me offline and my understanding is that we would not have the crypto command to beeline. It'd only be enabled for tests and thus not be a public API. bq. What I am trying to do with the crypto commands is that we can create encryption zones for specific tables during our tests. Users will have tables with different encryption zones, and I'd like to test different queries that work with these tables. If we can do this without exposing the command to the end user (i.e. only in tests), I think this is a good approach. My reasoning is: # We need to test db level encryption in addition to each table being a different encryption zone. # I feel like being able to create the ez's in the q-file is a little less error prone than creating the ez's in a separate location. It'd be very easy to mis-name the location of a table and thus not actually test with an encrypted location. Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[
https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252587#comment-14252587
]
Sergio Peña commented on HIVE-9167:
---
[~Ferd] [~brocknoland], I was thinking about how we could improve the .q test
files for encryption testing, and I think .q tests should be something like
this:
{noformat}
DROP TABLE IF EXISTS encrypted_table;
CREATE TABLE encrypted_table (key STRING, value STRING) LOCATION
'/user/hive/warehouse/encrypted_table';
crypto_helper create_key key1;
crypto_helper create_zone key1 /user/hive/warehouse/encrypted_table;
-- Test loading data from the local filesystem;
LOAD DATA LOCAL INPATH '../../data/files/kv1.txt' OVERWRITE INTO TABLE
encrypted_table;
SELECT * FROM encrypted_table;
-- Test loading data from the hdfs filesystem;
dfs -copyFromLocal ../../data/files/kv1.txt hdfs:///tmp/kv1.txt;
LOAD DATA INPATH '/tmp/kv1.txt' OVERWRITE INTO TABLE encrypted_table;
SELECT * FROM encrypted_table;
DROP TABLE encrypted_table;
crypto_helper delete_key key1;
{noformat}
The 'crypto_helper' is a command where we can use it into the .q files in order
to create keys zones on demand. This will also help us run some complex
INSERT queries with different encryption strength.
The above statements are just an example about how we can improve this testing
framework.
What do you think?
I am going to upload a patch for a partial review.
Enhance encryption testing framework to allow create keys zones inside .q
files
-
Key: HIVE-9167
URL: https://issues.apache.org/jira/browse/HIVE-9167
Project: Hive
Issue Type: Sub-task
Reporter: Sergio Peña
Assignee: Sergio Peña
The current implementation of the encryption testing framework on HIVE-8900
initializes a couple of encrypted databases to be used on .q test files. This
is useful in order to make tests small, but it does not test all details
found on the encryption implementation, such as: encrypted tables with
different encryption strength in the same database.
We need to allow this kind of encryption as it is how it will be used in the
real world where a database will have a few encrypted tables (not all the DB).
Also, we need to make this encryption framework flexible so that we can
create/delete keys zones on demand when running the .q files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252716#comment-14252716 ] Ferdinand Xu commented on HIVE-9167: Hi [~spena], do we real need add the crypto command for the beeline? We can use the explain statement to show the output directory(in quotes) EXPLAIN AUTHORIZATION query or add one more explain statement for the purpose. I think explain authorization will be OK for this purpose. At this point, the output for query test is empty and I am planing to file a jira for that. Any thoughts? https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
