Follow-up on HIVE-6992
Hi, Can someone review my patch uploaded for jira HIVE-6992? https://issues.apache.org/jira/browse/HIVE-6992 It is also available at review board at following link. https://reviews.apache.org/r/29448/
[jira] [Updated] (HIVE-6992) Implement PreparedStatement.getMetaData() in HIVE JDBC
[ https://issues.apache.org/jira/browse/HIVE-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prafulla T updated HIVE-6992: - Attachment: HIVE-6992.2.patch This patch includes fixes for NullpointerException found while testing. Implement PreparedStatement.getMetaData() in HIVE JDBC -- Key: HIVE-6992 URL: https://issues.apache.org/jira/browse/HIVE-6992 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bill Oliver Attachments: HIVE-6992.1.patch, HIVE-6992.2.patch It would be very helpful to have methods PreparedStatement.getMetaData() and also PreparedStatement.getParameterMetaData() implemented. I especially would like PreparedStatmeent.getMetaData() implemented, as I could prepare a SQL statement, and then get information about the result set, as well as information that the query is valid. I am pretty sure this information is available in some form. When you do an EXPLAIN query, the explain operation shows information about the result set including the column name/aliases and the column types. thanks you -bill -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6992) Implement PreparedStatement.getMetaData() in HIVE JDBC
[ https://issues.apache.org/jira/browse/HIVE-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prafulla T updated HIVE-6992: - Summary: Implement PreparedStatement.getMetaData() in HIVE JDBC (was: Implement PreparedStatement.getMetaData(), getParmeterMetaData()) Implement PreparedStatement.getMetaData() in HIVE JDBC -- Key: HIVE-6992 URL: https://issues.apache.org/jira/browse/HIVE-6992 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bill Oliver Attachments: HIVE-6992.1.patch It would be very helpful to have methods PreparedStatement.getMetaData() and also PreparedStatement.getParameterMetaData() implemented. I especially would like PreparedStatmeent.getMetaData() implemented, as I could prepare a SQL statement, and then get information about the result set, as well as information that the query is valid. I am pretty sure this information is available in some form. When you do an EXPLAIN query, the explain operation shows information about the result set including the column name/aliases and the column types. thanks you -bill -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9240) Implement PreparedStatement.getParmeterMetaData in hive JDBC
Prafulla T created HIVE-9240: Summary: Implement PreparedStatement.getParmeterMetaData in hive JDBC Key: HIVE-9240 URL: https://issues.apache.org/jira/browse/HIVE-9240 Project: Hive Issue Type: Task Components: JDBC Reporter: Prafulla T This ticket is created by splitting HIVE-6992. See HIVE-6992 for details -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 29448: HIVE-6992 - Support for PreparedStatement.getMetadata in hive-jdbc and server
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29448/ --- Review request for hive. Bugs: HIVE-6992 https://issues.apache.org/jira/browse/HIVE-6992 Repository: hive Description --- HIVE-6992 - Support for PreparedStatement.getMetadata in hive-jdbc and server This patch 1. Changes HiveSession/ICliService.executeStatement prototype to include two paramters. a. Boolean variable - prepareOnly to indicate if this execution request is only for preparing. b. existingOpHandle - to execute prepared operation. 2. Changes TExecuteStatementReq to support above two parameters. 3. Changes SQLOperation.java to support separate preparation step. 4. Adds new OperationState called PREPARED to indicate prepared operation. 5. Refactors HiveStatement.java/HiveQueryResult to support retrieving of schema from HiveStatement.java. 6. Changes HiveJdbc class to support PreparedStatement.getMetadata Also includes new (basic) unit-test for PreapredStatement.getMetadata Test for executeStatement will be added later after initial code-review I would like to know if my approach looks ok OR reviewers would like seperate PreapreStatement API like ExecuteStatement Diffs - trunk/itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 1647912 trunk/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2.java 1647912 trunk/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2SessionTimeout.java 1647912 trunk/jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 1647912 trunk/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 1647912 trunk/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 1647912 trunk/service/if/TCLIService.thrift 1647912 trunk/service/src/gen/thrift/gen-cpp/TCLIService_types.h 1647912 trunk/service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 1647912 trunk/service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 1647912 trunk/service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOperationState.java 1647912 trunk/service/src/gen/thrift/gen-py/TCLIService/ttypes.py 1647912 trunk/service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 1647912 trunk/service/src/java/org/apache/hive/service/cli/CLIService.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/ICLIService.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/OperationState.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/operation/MetadataOperation.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/operation/Operation.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/session/HiveSession.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 1647912 trunk/service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 1647912 trunk/service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 1647912 trunk/service/src/test/org/apache/hive/service/cli/operation/TestOperationLoggingAPI.java 1647912 trunk/service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 1647912 trunk/service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java 1647912 Diff: https://reviews.apache.org/r/29448/diff/ Testing --- Unit test added for PreparedStatement.getMetadata Also tested with simple JDBC program. Thanks, Prafulla
[jira] [Updated] (HIVE-6992) Implement PreparedStatement.getMetaData(), getParmeterMetaData()
[ https://issues.apache.org/jira/browse/HIVE-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prafulla T updated HIVE-6992: - Status: Patch Available (was: Open) Description of changes in attached patch HIVE-6992.1.patch HIVE-6992 - Support for PreparedStatement.getMetadata in hive-jdbc and server This patch 1. Changes HiveSession/ICliService.executeStatement prototype to include two paramters. a. Boolean variable - prepareOnly to indicate if this execution request is only for preparing. b. existingOpHandle - to execute prepared operation. 2. Changes TExecuteStatementReq to support above two parameters. 3. Changes SQLOperation.java to support separate preparation step. 4. Adds new OperationState called PREPARED to indicate prepared operation. 5. Refactors HiveStatement.java/HiveQueryResult to support retrieving of schema from HiveStatement.java. 6. Changes HiveJdbc class to support PreparedStatement.getMetadata Also includes new (basic) unit-test for PreapredStatement.getMetadata Test for executeStatement will be added later after initial code-review I would like to know if my approach looks ok OR reviewers would like seperate PreapreStatement API like ExecuteStatement Implement PreparedStatement.getMetaData(), getParmeterMetaData() Key: HIVE-6992 URL: https://issues.apache.org/jira/browse/HIVE-6992 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bill Oliver It would be very helpful to have methods PreparedStatement.getMetaData() and also PreparedStatement.getParameterMetaData() implemented. I especially would like PreparedStatmeent.getMetaData() implemented, as I could prepare a SQL statement, and then get information about the result set, as well as information that the query is valid. I am pretty sure this information is available in some form. When you do an EXPLAIN query, the explain operation shows information about the result set including the column name/aliases and the column types. thanks you -bill -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6992) Implement PreparedStatement.getMetaData(), getParmeterMetaData()
[ https://issues.apache.org/jira/browse/HIVE-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prafulla T updated HIVE-6992: - Attachment: HIVE-6992.1.patch Initial patch Implement PreparedStatement.getMetaData(), getParmeterMetaData() Key: HIVE-6992 URL: https://issues.apache.org/jira/browse/HIVE-6992 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bill Oliver Attachments: HIVE-6992.1.patch It would be very helpful to have methods PreparedStatement.getMetaData() and also PreparedStatement.getParameterMetaData() implemented. I especially would like PreparedStatmeent.getMetaData() implemented, as I could prepare a SQL statement, and then get information about the result set, as well as information that the query is valid. I am pretty sure this information is available in some form. When you do an EXPLAIN query, the explain operation shows information about the result set including the column name/aliases and the column types. thanks you -bill -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6992) Implement PreparedStatement.getMetaData(), getParmeterMetaData()
[ https://issues.apache.org/jira/browse/HIVE-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259813#comment-14259813 ] Prafulla T commented on HIVE-6992: -- This patch is also available for review on reviewboard at following link. https://reviews.apache.org/r/29448/ Implement PreparedStatement.getMetaData(), getParmeterMetaData() Key: HIVE-6992 URL: https://issues.apache.org/jira/browse/HIVE-6992 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bill Oliver Attachments: HIVE-6992.1.patch It would be very helpful to have methods PreparedStatement.getMetaData() and also PreparedStatement.getParameterMetaData() implemented. I especially would like PreparedStatmeent.getMetaData() implemented, as I could prepare a SQL statement, and then get information about the result set, as well as information that the query is valid. I am pretty sure this information is available in some form. When you do an EXPLAIN query, the explain operation shows information about the result set including the column name/aliases and the column types. thanks you -bill -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Need desig-review for HIVE-6992
Hi, I would like to contribute to Hive codebase by implementing PreparedStatement.getMetadata functionality requested in HIVE-6992. Can somebody review high level design idea that I proposed on Jira?
Test email
Test email for testing my subscription to the list.
[jira] [Commented] (HIVE-6992) Implement PreparedStatement.getMetaData(), getParmeterMetaData()
[ https://issues.apache.org/jira/browse/HIVE-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254997#comment-14254997 ] Prafulla T commented on HIVE-6992: -- I have some thoughts about how PraparedStatement.getMetadata can be implemented. Let me know your comments and I can go ahead and implement this functionality. h4. Here is my proposal. h5. On server-side: Add couple of parameters TExecuteStatementReq thrift * A bool variable prepareOnly indicate this is “prepare only” request. * An operation handle to indicate existing operation on server. Add couple of parameters to HiveSession.executeStatement and executeStatementAsy - operationHandle and prepareOnly. In HiveSessionImpl.executeStatementInternal method, f prepareOnly flag is set and Create operation and invoke “prepare” method instead of run. If ExecuteStatementReq has valid operationHandle ( non-null ), then use that operation instead of creating new operation. If prepareOnly flag is not set, Invoke run method on corresponding Operation. Return error if Operation(if given) is not in valid state. h5. On Client side: In PreparedStatement.getMetadata, Check if cached schema is available, if yes return that. Otherwise, send ExecuteStatementReq with prepareOnly flag and read schema using getSchema method. When this returns, Save operationHandle locally and use it for sending when user actually invokes PrepareMethod.execute* methods. Implement PreparedStatement.getMetaData(), getParmeterMetaData() Key: HIVE-6992 URL: https://issues.apache.org/jira/browse/HIVE-6992 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bill Oliver It would be very helpful to have methods PreparedStatement.getMetaData() and also PreparedStatement.getParameterMetaData() implemented. I especially would like PreparedStatmeent.getMetaData() implemented, as I could prepare a SQL statement, and then get information about the result set, as well as information that the query is valid. I am pretty sure this information is available in some form. When you do an EXPLAIN query, the explain operation shows information about the result set including the column name/aliases and the column types. thanks you -bill -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-7884) When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set
[ https://issues.apache.org/jira/browse/HIVE-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prafulla T resolved HIVE-7884. -- Resolution: Invalid We found that this was due to error in our program that fetches from hive. Resolving as Invalid. When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set -- Key: HIVE-7884 URL: https://issues.apache.org/jira/browse/HIVE-7884 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prafulla T In one of product of my company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only single column out of it, we get empty set instead of returning rows from partitions which match with single column used in partition filter. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7884) When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set
Prafulla T created HIVE-7884: Summary: When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set Key: HIVE-7884 URL: https://issues.apache.org/jira/browse/HIVE-7884 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prafulla T In one of product of our company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only in single column out of it, we get empty set instead of returning rows from partition which match with single column. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7884) When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set
[ https://issues.apache.org/jira/browse/HIVE-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prafulla T updated HIVE-7884: - Description: In one of product of my company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only in single column out of it, we get empty set instead of returning rows from partition which match with single column. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. was: In one of product of our company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only in single column out of it, we get empty set instead of returning rows from partition which match with single column. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set -- Key: HIVE-7884 URL: https://issues.apache.org/jira/browse/HIVE-7884 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prafulla T In one of product of my company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only in single column out of it, we get empty set instead of returning rows from partition which match with single column. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7884) When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set
[ https://issues.apache.org/jira/browse/HIVE-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prafulla T updated HIVE-7884: - Description: In one of product of my company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only single column out of it, we get empty set instead of returning rows from partitions which match with single column used in partition filter. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. was: In one of product of my company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only in single column out of it, we get empty set instead of returning rows from partition which match with single column. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. When partition filter containing single column from multiple partition is used in HCatInputFormat.setFilter, it returns empty set -- Key: HIVE-7884 URL: https://issues.apache.org/jira/browse/HIVE-7884 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prafulla T In one of product of my company, we use HCatInputFormat to import data from hadoop-hive to our database. We use HCatInputFormat.setFilter to pass partition filters based on partition columns. We see following issue in latest hive. When hive table has multiple partition columns and partition filter uses only single column out of it, we get empty set instead of returning rows from partitions which match with single column used in partition filter. This used to work earlier (hive 0.10.0 or 0.11.0), We experience this problem in hive 0.13.0. -- This message was sent by Atlassian JIRA (v6.2#6252)
Anyway in hive to measure query performance.
Hi, I was wondering if there is anyway in hive that can be used to measure the performance of variour components/operations of a single query run. Eg. Typecally query involvs various operations like tablescan, joins, aggregation, orderby etc. Can I get how much time was required for each of this? Also how do you measure hadoop-cluster performance as far as hive query/load run is concerned ? -- Best Regards, Prafulla V Tekawade
