[jira] [Resolved] (HIVE-13761) Select data from no Encrypt table where my cluste enable Encrypt
[ https://issues.apache.org/jira/browse/HIVE-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lushuai resolved HIVE-13761. Resolution: Not A Bug the other client close the filesystem. > Select data from no Encrypt table where my cluste enable Encrypt > -- > > Key: HIVE-13761 > URL: https://issues.apache.org/jira/browse/HIVE-13761 > Project: Hive > Issue Type: Bug > Components: Beeline, Encryption >Affects Versions: 1.2.1 >Reporter: lushuai > > 2016-05-14 14:43:35,962 WARN [HiveServer2-Handler-Pool: Thread-312]: > thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(492)) - Error > executing statement: > org.apache.hive.service.cli.HiveSQLException: Error while compiling > statement: FAILED: SemanticException Unable to determine if > hdfs://ns1/user/hive/warehouse/acl_db.db/tbl02 is encrypted: > java.io.IOException: Filesystem closed > at > org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) > at > org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:112) > at > org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:257) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Unable to > determine if hdfs://ns1/user/hive/warehouse/acl_db.db/tbl02 is encrypted: > java.io.IOException: Filesystem closed > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1868) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1545) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10077) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) > at > org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116) > at > org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110) > ... 15 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > determine if hdfs://ns1/user/hive/warehouse/acl_db.db/tbl02 is encrypted: > java.io.IOException: Filesystem closed > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1888) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:1965) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:1997) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1810) > ... 25 more > Caused by: java.io.IOException: Filesystem closed > at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:798) > at org.apache.hadoop.hdfs.DFSClient.getEZForPath(DFSClient.java:2966) > at > org.apache.hadoop.hdfs.Distributed
[jira] [Commented] (HIVE-13761) Select data from no Encrypt table where my cluste enable Encrypt
[ https://issues.apache.org/jira/browse/HIVE-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283453#comment-15283453 ] lushuai commented on HIVE-13761: use beeline to select table : select * from tbl02; Error: Error while compiling statement: FAILED: SemanticException Unable to determine if hdfs://ns1/user/hive/warehouse/acl_db.db/tbl02 is encrypted: java.io.IOExc but when I exit reconnect the beeline : !connect jdbc:hive2://host_name:1/default; it is ok. > Select data from no Encrypt table where my cluste enable Encrypt > -- > > Key: HIVE-13761 > URL: https://issues.apache.org/jira/browse/HIVE-13761 > Project: Hive > Issue Type: Bug > Components: Beeline, Encryption >Affects Versions: 1.2.1 >Reporter: lushuai > > 2016-05-14 14:43:35,962 WARN [HiveServer2-Handler-Pool: Thread-312]: > thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(492)) - Error > executing statement: > org.apache.hive.service.cli.HiveSQLException: Error while compiling > statement: FAILED: SemanticException Unable to determine if > hdfs://ns1/user/hive/warehouse/acl_db.db/tbl02 is encrypted: > java.io.IOException: Filesystem closed > at > org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) > at > org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:112) > at > org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:257) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Unable to > determine if hdfs://ns1/user/hive/warehouse/acl_db.db/tbl02 is encrypted: > java.io.IOException: Filesystem closed > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1868) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1545) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10077) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122) > at > org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116) > at > org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110) > ... 15 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > determine if hdfs://ns1/user/hive/warehouse/acl_db.db/tbl02 is encrypted: > java.io.IOException: Filesystem closed > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1888) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:1965) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:1997) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaD
[jira] [Commented] (HIVE-8065) Support HDFS encryption functionality on Hive
[
https://issues.apache.org/jira/browse/HIVE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271998#comment-15271998
]
lushuai commented on HIVE-8065:
---
How to create a table encryption table, for example, by specified DDL table
encryption attributes, implementation, and secure area bound.
By implementing MetaStoreEventListener in the onCreate Table, Drop Table,
onAlterTable, onAlterTable etc. and in combination with transparent encryption.
IS OK???
> Support HDFS encryption functionality on Hive
> -
>
> Key: HIVE-8065
> URL: https://issues.apache.org/jira/browse/HIVE-8065
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.13.1
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>
> The new encryption support on HDFS makes Hive incompatible and unusable when
> this feature is used.
> HDFS encryption is designed so that an user can configure different
> encryption zones (or directories) for multi-tenant environments. An
> encryption zone has an exclusive encryption key, such as AES-128 or AES-256.
> Because of security compliance, the HDFS does not allow to move/rename files
> between encryption zones. Renames are allowed only inside the same encryption
> zone. A copy is allowed between encryption zones.
> See HDFS-6134 for more details about HDFS encryption design.
> Hive currently uses a scratch directory (like /tmp/$user/$random). This
> scratch directory is used for the output of intermediate data (between MR
> jobs) and for the final output of the hive query which is later moved to the
> table directory location.
> If Hive tables are in different encryption zones than the scratch directory,
> then Hive won't be able to renames those files/directories, and it will make
> Hive unusable.
> To handle this problem, we can change the scratch directory of the
> query/statement to be inside the same encryption zone of the table directory
> location. This way, the renaming process will be successful.
> Also, for statements that move files between encryption zones (i.e. LOAD
> DATA), a copy may be executed instead of a rename. This will cause an
> overhead when copying large data files, but it won't break the encryption on
> Hive.
> Another security thing to consider is when using joins selects. If Hive joins
> different tables with different encryption key strengths, then the results of
> the select might break the security compliance of the tables. Let's say two
> tables with 128 bits and 256 bits encryption are joined, then the temporary
> results might be stored in the 128 bits encryption zone. This will conflict
> with the table encrypted with 256 bits temporary.
> To fix this, Hive should be able to select the scratch directory that is more
> secured/encrypted in order to save the intermediate data temporary with no
> compliance issues.
> For instance:
> {noformat}
> SELECT * FROM table-aes128 t1 JOIN table-aes256 t2 WHERE t1.id == t2.id;
> {noformat}
> - This should use a scratch directory (or staging directory) inside the
> table-aes256 table location.
> {noformat}
> INSERT OVERWRITE TABLE table-unencrypted SELECT * FROM table-aes1;
> {noformat}
> - This should use a scratch directory inside the table-aes1 location.
> {noformat}
> FROM table-unencrypted
> INSERT OVERWRITE TABLE table-aes128 SELECT id, name
> INSERT OVERWRITE TABLE table-aes256 SELECT id, name
> {noformat}
> - This should use a scratch directory on each of the tables locations.
> - The first SELECT will have its scratch directory on table-aes128 directory.
> - The second SELECT will have its scratch directory on table-aes256 directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9264) Merge encryption branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271977#comment-15271977 ] lushuai commented on HIVE-9264: --- How to create a table encryption table, for example, by specified DDL table encryption attributes, implementation, and secure area bound. By implementing MetaStoreEventListener in the onCreate Table, Drop Table, onAlterTable, onAlterTable etc. and in combination with transparent encryption. IS OK??? > Merge encryption branch to trunk > > > Key: HIVE-9264 > URL: https://issues.apache.org/jira/browse/HIVE-9264 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.15.0 >Reporter: Brock Noland >Assignee: Brock Noland > Labels: TODOC15 > Fix For: 1.1.0 > > Attachments: HIVE-9264.1.patch, HIVE-9264.2.patch, HIVE-9264.2.patch, > HIVE-9264.2.patch, HIVE-9264.3.patch, HIVE-9264.3.patch, HIVE-9264.3.patch, > HIVE-9264.addendum.patch > > > The team working on the encryption branch would like to merge their work to > trunk. This jira will track that effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
