[jira] [Created] (HIVE-6840) Use Unordered Output for Bucket Map Joins on Tez
Siddharth Seth created HIVE-6840: Summary: Use Unordered Output for Bucket Map Joins on Tez Key: HIVE-6840 URL: https://issues.apache.org/jira/browse/HIVE-6840 Project: Hive Issue Type: Bug Reporter: Siddharth Seth Assignee: Siddharth Seth Tez 0.4 adds a placeholder UnorderedOutput. Once Hive is changed to use 0.4, it should be possible to make use of this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6840) Use Unordered Output for Bucket Map Joins on Tez
[ https://issues.apache.org/jira/browse/HIVE-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-6840: - Attachment: HIVE-6840.1.patch Simple patch. Requires tez 0.4 dependency though, so will not compile at the moment. [~vikram.dixit], [~hagleitn] - could you please review. Thanks Use Unordered Output for Bucket Map Joins on Tez Key: HIVE-6840 URL: https://issues.apache.org/jira/browse/HIVE-6840 Project: Hive Issue Type: Bug Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HIVE-6840.1.patch Tez 0.4 adds a placeholder UnorderedOutput. Once Hive is changed to use 0.4, it should be possible to make use of this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6837: --- Affects Version/s: 0.13.0 HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Labels: HiveServer2 Fix For: 0.13.0 Attachments: hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6837: --- Fix Version/s: 0.13.0 HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Labels: HiveServer2 Fix For: 0.13.0 Attachments: hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959682#comment-13959682 ] Vaibhav Gumashta commented on HIVE-6837: [~taksaito] Thanks for the log and additional testing info Tak! This is caused by threadlocals lying all over the place. There are 2 set of threadlocals for storing ip address, usernames (one in TSetIpAddressProcessor and the other in HadoopThriftAuthBridge20S.Server). I think it makes sense to access those through one interface, otherwise similar problems will pop up again. HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Labels: HiveServer2 Fix For: 0.13.0 Attachments: hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6837: --- Labels: HiveServer2 (was: ) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Labels: HiveServer2 Fix For: 0.13.0 Attachments: hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6837: --- Labels: (was: HiveServer2) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959684#comment-13959684 ] Vaibhav Gumashta commented on HIVE-6837: [~rhbutani] [~thejas] This is a bug for 13. I'll have a patch ready by tomorrow. HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
Jitendra Nath Pandey created HIVE-6841: -- Summary: Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6827) Disable insecure commands with std sql auth
[ https://issues.apache.org/jira/browse/HIVE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959701#comment-13959701 ] Thejas M Nair commented on HIVE-6827: - This patch is causing a unit test failure(TestCommandProcessorFactory.testAvailableCommands). I am thinking of reverting it for now. Looks like a genuine unit test failure actually got ignored. Disable insecure commands with std sql auth --- Key: HIVE-6827 URL: https://issues.apache.org/jira/browse/HIVE-6827 Project: Hive Issue Type: Task Components: Authorization, Security Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6827.patch Disable insecure command on auth V2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6332) HCatConstants Documentation needed
[ https://issues.apache.org/jira/browse/HIVE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959704#comment-13959704 ] Lefty Leverenz commented on HIVE-6332: -- [~sushanth], you could flesh out the introduction with instructions on how/where/when to set these properties. If they shouldn't be set by users, you could say they're generally set by administrators. A simple example or two would be helpful. Right now I'm documenting hcat.dynamic.partitioning.custom.pattern (HIVE-6109) but it isn't much use without information about how to set it. The jira description calls it a job config -- does that mean it can be set for a single CREATE/ALTER TABLE statement? Is that generally true of HCatConstants configs? HCatConstants Documentation needed -- Key: HIVE-6332 URL: https://issues.apache.org/jira/browse/HIVE-6332 Project: Hive Issue Type: Task Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan HCatConstants documentation is near non-existent, being defined only as comments in code for the various parameters. Given that a lot of api winds up being implemented as knobs that can be tweaked here, we should have a public facing doc for this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6827) Disable insecure commands with std sql auth
[ https://issues.apache.org/jira/browse/HIVE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959706#comment-13959706 ] Thejas M Nair commented on HIVE-6827: - I have reverted it to fix the unit test failure. Disable insecure commands with std sql auth --- Key: HIVE-6827 URL: https://issues.apache.org/jira/browse/HIVE-6827 Project: Hive Issue Type: Task Components: Authorization, Security Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6827.patch Disable insecure command on auth V2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6838) q.out files need correction for stats properties - sample8,transform_ppr1,transform_ppr2,union_ppr
[ https://issues.apache.org/jira/browse/HIVE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959708#comment-13959708 ] Jitendra Nath Pandey commented on HIVE-6838: +1 q.out files need correction for stats properties - sample8,transform_ppr1,transform_ppr2,union_ppr -- Key: HIVE-6838 URL: https://issues.apache.org/jira/browse/HIVE-6838 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6838.1.patch HIVE-6808 updated stats information q.out of the following test files, causing failures - sample8,transform_ppr1,transform_ppr2,union_ppr . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6838) q.out files need correction for stats properties - sample8,transform_ppr1,transform_ppr2,union_ppr
[ https://issues.apache.org/jira/browse/HIVE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959709#comment-13959709 ] Thejas M Nair commented on HIVE-6838: - Committing it shortly to avoid the false alarms in precommit tests. q.out files need correction for stats properties - sample8,transform_ppr1,transform_ppr2,union_ppr -- Key: HIVE-6838 URL: https://issues.apache.org/jira/browse/HIVE-6838 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6838.1.patch HIVE-6808 updated stats information q.out of the following test files, causing failures - sample8,transform_ppr1,transform_ppr2,union_ppr . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6838) q.out files need correction for stats properties - sample8,transform_ppr1,transform_ppr2,union_ppr
[ https://issues.apache.org/jira/browse/HIVE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6838: Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) q.out files need correction for stats properties - sample8,transform_ppr1,transform_ppr2,union_ppr -- Key: HIVE-6838 URL: https://issues.apache.org/jira/browse/HIVE-6838 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6838.1.patch HIVE-6808 updated stats information q.out of the following test files, causing failures - sample8,transform_ppr1,transform_ppr2,union_ppr . -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 13845: HIVE-5155: Support secure proxy user access to HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13845/#review39522 --- service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java https://reviews.apache.org/r/13845/#comment71937 What does (hiveAuthFactory == null) mean? - Vaibhav Gumashta On March 10, 2014, 6:39 p.m., Prasad Mujumdar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13845/ --- (Updated March 10, 2014, 6:39 p.m.) Review request for hive, Brock Noland, Carl Steinbach, and Thejas Nair. Bugs: HIVE-5155 https://issues.apache.org/jira/browse/HIVE-5155 Repository: hive-git Description --- Delegation token support - Enable delegation token connection for HiveServer2 Enhance the TCLIService interface to support delegation token requests Support passing the delegation token connection type via JDBC URL and Beeline option Direct proxy access - Define new proxy user property Shim interfaces to validate proxy access for a given user Note that the diff doesn't include thrift generated code. Diffs - beeline/pom.xml 02bfaaa beeline/src/java/org/apache/hive/beeline/BeeLine.java e63a3b0 beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 91e20ec beeline/src/java/org/apache/hive/beeline/Commands.java d2d7fd3 beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 94178ef beeline/src/test/org/apache/hive/beeline/ProxyAuthTest.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38 conf/hive-default.xml.template a5a85b4 data/files/ProxyAuth.res PRE-CREATION itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 8210e75 jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java d08e05b jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 607fc7a jdbc/src/java/org/apache/hive/jdbc/Utils.java 608837e service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java d8ba3aa service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 71dc592 service/src/java/org/apache/hive/service/auth/PlainSaslHelper.java 15b1675 service/src/java/org/apache/hive/service/cli/CLIService.java 2b1e712 service/src/java/org/apache/hive/service/cli/CLIServiceClient.java b9d1489 service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java a31ea94 service/src/java/org/apache/hive/service/cli/ICLIService.java 621d689 service/src/java/org/apache/hive/service/cli/session/HiveSession.java c8fb8ec service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java d6d0d27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java b934ebe service/src/java/org/apache/hive/service/cli/session/SessionManager.java cec3b04 service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 26bda5a service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 3675e86 service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 8fa4afd service/src/test/org/apache/hive/service/cli/session/TestSessionHooks.java 2fac800 shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 51c8051 shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java e205caa shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java 29114f0 shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java dc89de1 shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java e15ab4e shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 03f4e51 Diff: https://reviews.apache.org/r/13845/diff/ Testing --- Since this requires kerberos setup, its tested by a standalone test program that runs various existing and new secure connection scenarios. The test code is attached to the ticket at https://issues.apache.org/jira/secure/attachment/12600119/ProxyAuth.java Thanks, Prasad Mujumdar
[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959723#comment-13959723 ] Vaibhav Gumashta commented on HIVE-5155: [~prasadm] The relevant question is regarding the intent of (hiveAuthFactory == null) in ThriftCLIService. Thanks! Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.13.0 Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155-noThrift.7.patch, HIVE-5155-noThrift.8.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, HIVE-5155.4.patch, HIVE-5155.5.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959722#comment-13959722 ] Vaibhav Gumashta commented on HIVE-5155: Hi [~prasadm], I had a question: https://reviews.apache.org/r/13845/#comment71937. Thanks! Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.13.0 Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155-noThrift.7.patch, HIVE-5155-noThrift.8.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, HIVE-5155.4.patch, HIVE-5155.5.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6841: --- Status: Patch Available (was: Open) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6841: --- Attachment: HIVE-6841.1.patch Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19789: HIVE-6739 Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19789/#review39521 --- ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java https://reviews.apache.org/r/19789/#comment71934 Document all the input params/just rename the params to reflect their use. hdfsPathStr is the destination directory right? ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java https://reviews.apache.org/r/19789/#comment71932 Actually looks like this method localizes an array of files. ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java https://reviews.apache.org/r/19789/#comment71933 Definitely needs method(s) renaming. ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java https://reviews.apache.org/r/19789/#comment71936 Open a jira for this. ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java https://reviews.apache.org/r/19789/#comment71938 Aren't you walking the work graph here to populate the job conf? - Vikram Dixit Kumaraswamy On April 3, 2014, 2:26 a.m., Sergey Shelukhin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19789/ --- (Updated April 3, 2014, 2:26 a.m.) Review request for hive, Gunther Hagleitner and Vikram Dixit Kumaraswamy. Repository: hive-git Description --- See jira Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 78e965b ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 720b8d5 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 5f0f353 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 385e22e ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fdbd996 ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 38c4c11 ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java e1cc3f4 ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java 996268e ql/src/java/org/apache/hadoop/hive/ql/plan/UnionWork.java 60781e6 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 93eccf8 ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java d2c332c ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 5ad4250 ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 4810fd0 Diff: https://reviews.apache.org/r/19789/diff/ Testing --- Thanks, Sergey Shelukhin
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959750#comment-13959750 ] Lefty Leverenz commented on HIVE-6109: -- Please review and correct the doc here: * [HCatalog Dynamic Partitioning: External Tables |https://cwiki.apache.org/confluence/display/Hive/HCatalog+DynamicPartitions#HCatalogDynamicPartitions-ExternalTables] Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Fix For: 0.13.0 Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[
https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959762#comment-13959762
]
Hive QA commented on HIVE-6739:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638400/HIVE-6739.02.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5547 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
org.apache.hadoop.hive.ql.processors.TestCommandProcessorFactory.testAvailableCommands
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2104/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2104/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638400
Hive HBase query fails on Tez due to missing jars and then due to NPE in
getSplits
--
Key: HIVE-6739
URL: https://issues.apache.org/jira/browse/HIVE-6739
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: 0.13.0
Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.patch,
HIVE-6739.preliminary.patch
Tez paths in Hive never call configure on the input/output operators, so
(among other things, potentially) requisite files never get added to the job
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6447) Bucket map joins in hive-tez
[
https://issues.apache.org/jira/browse/HIVE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959765#comment-13959765
]
Vikram Dixit K commented on HIVE-6447:
--
Hi Lefty,
This would be the change to the template file:
{code}
property
namehive.convert.join.bucket.mapjoin.tez/name
valuefalse/value
descriptionWhether joins can be automatically converted to bucket map joins
in hive when tez is used as the execution engine./description
/property
{code}
Thanks
Vikram.
Bucket map joins in hive-tez
Key: HIVE-6447
URL: https://issues.apache.org/jira/browse/HIVE-6447
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: tez-branch
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6447.1.patch, HIVE-6447.10.patch,
HIVE-6447.11.patch, HIVE-6447.12.patch, HIVE-6447.13.patch,
HIVE-6447.2.patch, HIVE-6447.3.patch, HIVE-6447.4.patch, HIVE-6447.5.patch,
HIVE-6447.6.patch, HIVE-6447.7.patch, HIVE-6447.8.patch, HIVE-6447.9.patch,
HIVE-6447.WIP.patch
Support bucket map joins in tez.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6720) Implement getURL()
[ https://issues.apache.org/jira/browse/HIVE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959767#comment-13959767 ] Kang commented on HIVE-6720: I want to watch the received error message from hive. Implement getURL() --- Key: HIVE-6720 URL: https://issues.apache.org/jira/browse/HIVE-6720 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.12.0 Reporter: Jonathan Seidman Priority: Minor DatabaseMetaData.getURL() throws an unsupported exception. This should be modified to return a valid value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959769#comment-13959769 ] Satish Mittal commented on HIVE-6109: - [~leftylev], Very nicely documented, with detailed examples that explain the feature! A minor comment: the number of partitions is not consistent in the 1st example. The schema has 6 partitions (year, month, day, hour, minute, country), but examples have 4/5 partitions. A new user might get confused. Would be good to be consistent throughout. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Fix For: 0.13.0 Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5687) Streaming support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated HIVE-5687: -- Attachment: (was: HIVE-5687.v5.patch) Streaming support in Hive - Key: HIVE-5687 URL: https://issues.apache.org/jira/browse/HIVE-5687 Project: Hive Issue Type: Sub-task Reporter: Roshan Naik Assignee: Roshan Naik Labels: ACID, Streaming Fix For: 0.13.0 Attachments: 5687-api-spec4.pdf, 5687-draft-api-spec.pdf, 5687-draft-api-spec2.pdf, 5687-draft-api-spec3.pdf, HIVE-5687-unit-test-fix.patch, HIVE-5687.patch, HIVE-5687.v2.patch, HIVE-5687.v3.patch, HIVE-5687.v4.patch, Hive Streaming Ingest API for v3 patch.pdf, Hive Streaming Ingest API for v4 patch.pdf Implement support for Streaming data into HIVE. - Provide a client streaming API - Transaction support: Clients should be able to periodically commit a batch of records atomically - Immediate visibility: Records should be immediately visible to queries on commit - Should not overload HDFS with too many small files Use Cases: - Streaming logs into HIVE via Flume - Streaming results of computations from Storm -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5687) Streaming support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated HIVE-5687: -- Attachment: HIVE-5687.v5.patch refreshing patch v5 with minor fix to compile with hadoop1 profile Streaming support in Hive - Key: HIVE-5687 URL: https://issues.apache.org/jira/browse/HIVE-5687 Project: Hive Issue Type: Sub-task Reporter: Roshan Naik Assignee: Roshan Naik Labels: ACID, Streaming Fix For: 0.13.0 Attachments: 5687-api-spec4.pdf, 5687-draft-api-spec.pdf, 5687-draft-api-spec2.pdf, 5687-draft-api-spec3.pdf, HIVE-5687-unit-test-fix.patch, HIVE-5687.patch, HIVE-5687.v2.patch, HIVE-5687.v3.patch, HIVE-5687.v4.patch, HIVE-5687.v5.patch, Hive Streaming Ingest API for v3 patch.pdf, Hive Streaming Ingest API for v4 patch.pdf Implement support for Streaming data into HIVE. - Provide a client streaming API - Transaction support: Clients should be able to periodically commit a batch of records atomically - Immediate visibility: Records should be immediately visible to queries on commit - Should not overload HDFS with too many small files Use Cases: - Streaming logs into HIVE via Flume - Streaming results of computations from Storm -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6839) Config changes in hive CLI should be propagated to tez configs in Hive
[ https://issues.apache.org/jira/browse/HIVE-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-6839: --- Summary: Config changes in hive CLI should be propagated to tez configs in Hive (was: SetProcessor should propagate config changes to TezSessionState) Config changes in hive CLI should be propagated to tez configs in Hive -- Key: HIVE-6839 URL: https://issues.apache.org/jira/browse/HIVE-6839 Project: Hive Issue Type: Bug Components: Tez Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical When hive.execution.engine=tez is enabled, hive cli launches a tez session. But any config changes via set command is not propagated to TezSessions. E.g 1. hive --hiveconfig hive.execution.engine=tez 2. set tez.runtime.io.sort.mb = 1024; 3. set tez.runtime.io.sort.mb (//to verify if the parameter is set) 4. Run a hive query and check the app logs. It would not reflect 1024 for tez.runtime.io.sort.mb -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6809) Support bulk deleting directories for partition drop with partial spec
[
https://issues.apache.org/jira/browse/HIVE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959849#comment-13959849
]
Hive QA commented on HIVE-6809:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638396/HIVE-6809.2.patch.txt
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5547 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_partitions_partialspec
org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testDropPartitionFail1
org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testDropPartitionFail2
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2106/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2106/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638396
Support bulk deleting directories for partition drop with partial spec
--
Key: HIVE-6809
URL: https://issues.apache.org/jira/browse/HIVE-6809
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: HIVE-6809.1.patch.txt, HIVE-6809.2.patch.txt
In busy hadoop system, dropping many of partitions takes much more time than
expected. In hive-0.11.0, removing 1700 partitions by single partial spec
took 90 minutes, which is reduced to 3 minutes when deleteData is set false.
I couldn't test this in recent hive, which has HIVE-6256 but if the
time-taking part is mostly from removing directories, it seemed not helpful
to reduce whole processing time.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6824) Hive HBase query fails on Tez due to missing jars - part 2
[
https://issues.apache.org/jira/browse/HIVE-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959850#comment-13959850
]
Hive QA commented on HIVE-6824:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638401/HIVE-6824.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2108/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2108/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-2108/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java'
Reverted
'metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java'
Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java'
Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py'
Reverted
'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote'
Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp'
Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h'
Reverted
'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp'
Reverted 'metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb'
Reverted
'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java'
Reverted 'metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php'
Reverted 'metastore/if/hive_metastore.thrift'
Reverted
'ql/src/test/results/clientnegative/drop_partition_filter_failure.q.out'
Reverted 'ql/src/test/results/clientnegative/drop_partition_failure.q.out'
Reverted 'ql/src/test/results/clientpositive/drop_multi_partitions.q.out'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target
itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-unit/target itests/custom-serde/target
itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target
hcatalog/server-extensions/target hcatalog/core/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen
contrib/target service/target serde/target beeline/target odbc/target
cli/target ql/dependency-reduced-pom.xml ql/target
ql/src/test/results/clientnegative/drop_partition_failure.q.out.orig
ql/src/test/results/clientnegative/drop_partition_filter_failure.q.out.orig
ql/src/test/results/clientpositive/drop_multi_partitions.q.out.orig
ql/src/test/results/clientpositive/drop_partitions_partialspec.q.out
ql/src/test/queries/clientpositive/drop_partitions_partialspec.q
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java.orig
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1584603.
At revision 1584603.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+
[jira] [Resolved] (HIVE-6839) Config changes in hive CLI should be propagated to tez configs in Hive
[ https://issues.apache.org/jira/browse/HIVE-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved HIVE-6839. Resolution: Invalid It appears that tez.runtime.io.sort.mb can not be set from CLI. It gets unset automatically in the job by Tez's MultiStageMRConfToTezTranslator. However, as per CLI's output it would still retain the newer value . It would be good to print some message from tez, whenever these values are unset. Since this is not hive related issue, I am closing this ticket as invalid Config changes in hive CLI should be propagated to tez configs in Hive -- Key: HIVE-6839 URL: https://issues.apache.org/jira/browse/HIVE-6839 Project: Hive Issue Type: Bug Components: Tez Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Critical When hive.execution.engine=tez is enabled, hive cli launches a tez session. But any config changes via set command is not propagated to TezSessions. E.g 1. hive --hiveconfig hive.execution.engine=tez 2. set tez.runtime.io.sort.mb = 1024; 3. set tez.runtime.io.sort.mb (//to verify if the parameter is set) 4. Run a hive query and check the app logs. It would not reflect 1024 for tez.runtime.io.sort.mb -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[
https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959905#comment-13959905
]
Hive QA commented on HIVE-6782:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638617/HIVE-6782.5.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5546 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.ql.exec.tez.TestTezSessionPool.testGetNonDefaultSession
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2109/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2109/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638617
HiveServer2Concurrency issue when running with tez intermittently, throwing
org.apache.tez.dag.api.SessionNotRunning: Application not running error
-
Key: HIVE-6782
URL: https://issues.apache.org/jira/browse/HIVE-6782
Project: Hive
Issue Type: Bug
Components: Tez
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch,
HIVE-6782.4.patch, HIVE-6782.5.patch
HiveServer2 concurrency is failing intermittently when using tez, throwing
org.apache.tez.dag.api.SessionNotRunning: Application not running error
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6826) Hive-tez has issues when different partitions work off of different input types
[
https://issues.apache.org/jira/browse/HIVE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959993#comment-13959993
]
Hive QA commented on HIVE-6826:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638420/HIVE-6826.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5546 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2110/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2110/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638420
Hive-tez has issues when different partitions work off of different input
types
---
Key: HIVE-6826
URL: https://issues.apache.org/jira/browse/HIVE-6826
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.13.0, 0.14.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Attachments: HIVE-6826.1.patch
create table test (key int, value string) partitioned by (p int) stored as
textfile;
insert into table test partition (p=1) select * from src limit 10;
alter table test set fileformat orc;
insert into table test partition (p=2) select * from src limit 10;
describe test;
select * from test where p=1 and key 0;
select * from test where p=2 and key 0;
select * from test where key 0;
throws a classcast exception
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5820) Neither avro.schema.literal nor avro.schema.url specified, can't determine table schema
[
https://issues.apache.org/jira/browse/HIVE-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960042#comment-13960042
]
Wayne Zhu commented on HIVE-5820:
-
I have the same issue on CDH 4.3.0.
$ rpm -qa | grep hive
hive-hbase-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch
hive-metastore-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch
hive-jdbc-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch
hive-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch
Here is Hive steps:
hive CREATE TABLE avro_price_internal
PARTITIONED BY (year int, month int, day int, hour int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs:///user/wzhu00/camus/camus-avsc/PriceSchema.avsc'
);
OK
Time taken: 2.762 seconds
hive ALTER TABLE avro_price_internal ADD IF NOT EXISTS PARTITION (year=2014,
month=03, day=31, hour=19) LOCATION
'/user/wzhu00/camus/dest/pricesingle/hourly/2014/03/31/19';
OK
Time taken: 0.595 seconds
hive select * from avro_price_internal;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.serde2.avro.BadSchemaException
Time taken: 0.463 seconds
Neither avro.schema.literal nor avro.schema.url specified, can't determine
table schema
---
Key: HIVE-5820
URL: https://issues.apache.org/jira/browse/HIVE-5820
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Environment: CDH 4.3 Hive 0.10.0+121
Reporter: Sergey
Hi, we've created a table:
{code}
create table tmp
comment 'tmp'
partitioned by (year string, month string, day string, fulldate string)
row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
stored as
inputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
location '/user/lol/tmp'
tblproperties ('avro.schema.literal' =
'{name: tmp, doc: version 0.0.1, type: record, fields: [
{name: a, type: int},
{name: b, type: int}
]}'
)
{code}
And we try to query it:
{code}
select * from tmp
{code}
and we get an exception
{code}
13/11/14 17:12:15 WARN avro.AvroSerdeUtils: Encountered AvroSerdeException
determining schema. Returning signal schema to indicate problem
org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Neither
avro.schema.literal nor avro.schema.url specified, can't determine table
schema
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:66)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:249)
at
org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:251)
at
org.apache.hadoop.hive.ql.metadata.Partition.initialize(Partition.java:217)
at
org.apache.hadoop.hive.ql.metadata.Partition.init(Partition.java:107)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:1573)
at
org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:190)
at
org.apache.hadoop.hive.ql.parse.ParseContext.getPrunedPartitions(ParseContext.java:561)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.checkTree(SimpleFetchOptimizer.java:144)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:74)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:102)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8200)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.checkedCompile(BeeswaxServiceImpl.java:247)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.compile(BeeswaxServiceImpl.java:200)
at
com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:830)
at
com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:823)
at java.security.AccessController.doPrivileged(Native Method)
[jira] [Commented] (HIVE-5820) Neither avro.schema.literal nor avro.schema.url specified, can't determine table schema
[
https://issues.apache.org/jira/browse/HIVE-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960046#comment-13960046
]
Sergey commented on HIVE-5820:
--
Can you post your schema?
Neither avro.schema.literal nor avro.schema.url specified, can't determine
table schema
---
Key: HIVE-5820
URL: https://issues.apache.org/jira/browse/HIVE-5820
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Environment: CDH 4.3 Hive 0.10.0+121
Reporter: Sergey
Hi, we've created a table:
{code}
create table tmp
comment 'tmp'
partitioned by (year string, month string, day string, fulldate string)
row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
stored as
inputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
location '/user/lol/tmp'
tblproperties ('avro.schema.literal' =
'{name: tmp, doc: version 0.0.1, type: record, fields: [
{name: a, type: int},
{name: b, type: int}
]}'
)
{code}
And we try to query it:
{code}
select * from tmp
{code}
and we get an exception
{code}
13/11/14 17:12:15 WARN avro.AvroSerdeUtils: Encountered AvroSerdeException
determining schema. Returning signal schema to indicate problem
org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Neither
avro.schema.literal nor avro.schema.url specified, can't determine table
schema
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:66)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:249)
at
org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:251)
at
org.apache.hadoop.hive.ql.metadata.Partition.initialize(Partition.java:217)
at
org.apache.hadoop.hive.ql.metadata.Partition.init(Partition.java:107)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:1573)
at
org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:190)
at
org.apache.hadoop.hive.ql.parse.ParseContext.getPrunedPartitions(ParseContext.java:561)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.checkTree(SimpleFetchOptimizer.java:144)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:74)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:102)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8200)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.checkedCompile(BeeswaxServiceImpl.java:247)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.compile(BeeswaxServiceImpl.java:200)
at
com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:830)
at
com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:823)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772)
at
com.cloudera.beeswax.BeeswaxServiceImpl.query(BeeswaxServiceImpl.java:822)
at
com.cloudera.beeswax.api.BeeswaxService$Processor$query.getResult(BeeswaxService.java:915)
at
com.cloudera.beeswax.api.BeeswaxService$Processor$query.getResult(BeeswaxService.java:899)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
13/11/14 17:12:15 INFO parse.SemanticAnalyzer: Completed plan generation
13/11/14 17:12:15
[jira] [Commented] (HIVE-6840) Use Unordered Output for Bucket Map Joins on Tez
[ https://issues.apache.org/jira/browse/HIVE-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960073#comment-13960073 ] Gunther Hagleitner commented on HIVE-6840: -- +1 LGTM. Can you switch the pom to 0.4.0 in the same patch? Since it's needed for the patch. Use Unordered Output for Bucket Map Joins on Tez Key: HIVE-6840 URL: https://issues.apache.org/jira/browse/HIVE-6840 Project: Hive Issue Type: Bug Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HIVE-6840.1.patch Tez 0.4 adds a placeholder UnorderedOutput. Once Hive is changed to use 0.4, it should be possible to make use of this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6828) Hive tez bucket map join conversion interferes with map join conversion
[
https://issues.apache.org/jira/browse/HIVE-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960090#comment-13960090
]
Hive QA commented on HIVE-6828:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638424/HIVE-6828.1.patch
{color:green}SUCCESS:{color} +1 5518 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2112/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2112/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638424
Hive tez bucket map join conversion interferes with map join conversion
---
Key: HIVE-6828
URL: https://issues.apache.org/jira/browse/HIVE-6828
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.13.0, 0.14.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Attachments: HIVE-6828.1.patch
The issue is that bucket count is used for checking the scaled down size of
the hash tables but is used later on to convert to the map join as well which
may be incorrect in cases where the entire hash table does not fit in the
specified size.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5820) Neither avro.schema.literal nor avro.schema.url specified, can't determine table schema
[
https://issues.apache.org/jira/browse/HIVE-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960093#comment-13960093
]
Wayne Zhu commented on HIVE-5820:
-
Sergy,
Thanks for looking into it again. Here is the schema.
{
namespace:greenbox.ecom.sears.com,
type:record,
name:PriceSchema,
fields:[
{
name:price_id,
type:string,
default :
},
{
name:price_programType,
type:string,
default :
},
{
name:price_sale_salePrice,
type:float,
default : 0
},
{
name:price_sale_saleEndDate,
type:string,
default :
},
{
name:price_regPrice,
type:float,
default : 0
},
{
name:price_promotext,
type:string,
default :
},
{
name:price_mapVal,
type:float,
default : 0
},
{
name:price_shipping_minRate,
type: float,
default : 0
},
{
name:price_shipping_isSywrMaxShipElig,
type: boolean,
default : false
},
{
name:price_shipping_handlingFee,
type:float,
default : 0
},
{
name:price_shipping_mode_exp_excluded,
type:string,
default :
},
{
name:price_shipping_mode_exp_price,
type:float,
default : 0
},
{
name:price_shipping_mode_gnd_excluded,
type:string,
default :
},
{
name:price_shipping_mode_gnd_free_startDt,
type:string,
default :
},
{
name:price_shipping_mode_gnd_free_endDt,
type:string,
default :
},
{
name:price_shipping_mode_gnd_free_promoTxt,
type:string,
default :
},
{
name:price_shipping_mode_gnd_free_price,
type: boolean,
default : false
},
{
name:price_shipping_mode_prem_excluded,
type:string,
default :
},
{
name:price_shipping_mode_prem_price,
type:float,
default : 0
},
{
name:price_handlingFee,
type: boolean,
default : false
},
{
name:price_meta_createdTs,
type:string,
default :
},
{
name:price_meta_lastModifiedBy,
type:string,
default :
},
{
name:price_meta_modifiedTs,
type:string,
default :
},
{
name:price_meta_schemaVer,
type:string,
default :
},
{
name:price_isCENotExist,
type: boolean,
default : false
}
]
}
The table without partition works fine.
Neither avro.schema.literal nor avro.schema.url specified, can't determine
table schema
---
Key: HIVE-5820
URL: https://issues.apache.org/jira/browse/HIVE-5820
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Environment: CDH 4.3
[jira] [Updated] (HIVE-6743) Allow specifying the log level for Tez tasks
[ https://issues.apache.org/jira/browse/HIVE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-6743: - Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk and branch. Thanks [~sseth]! Allow specifying the log level for Tez tasks Key: HIVE-6743 URL: https://issues.apache.org/jira/browse/HIVE-6743 Project: Hive Issue Type: Improvement Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.13.0 Attachments: HIVE-6743.1.patch, HIVE-6743.2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5820) Neither avro.schema.literal nor avro.schema.url specified, can't determine table schema
[
https://issues.apache.org/jira/browse/HIVE-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960107#comment-13960107
]
Wayne Zhu commented on HIVE-5820:
-
It is working now when I removed the data from the partition.
hive select * from avro_price_external limit 1;
OK
Time taken: 0.245 seconds
So something is wrong when Hive tries to read the data.
Here is what I tried with the external table.
CREATE EXTERNAL TABLE avro_price_external
PARTITIONED BY (year int, month int, day int, hour int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs:///user/wzhu00/camus/camus-avsc/PriceSchema.avsc'
);
hive ALTER TABLE avro_price_external ADD IF NOT EXISTS PARTITION (year=2014,
month=03, day=31, hour=19) LOCATION
'/user/wzhu00/camus/dest/pricesingle/hourly/2014/03/31/19';
hive select * from avro_price_external;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.serde2.avro.BadSchemaException
Time taken: 0.301 seconds
Neither avro.schema.literal nor avro.schema.url specified, can't determine
table schema
---
Key: HIVE-5820
URL: https://issues.apache.org/jira/browse/HIVE-5820
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Environment: CDH 4.3 Hive 0.10.0+121
Reporter: Sergey
Hi, we've created a table:
{code}
create table tmp
comment 'tmp'
partitioned by (year string, month string, day string, fulldate string)
row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
stored as
inputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
location '/user/lol/tmp'
tblproperties ('avro.schema.literal' =
'{name: tmp, doc: version 0.0.1, type: record, fields: [
{name: a, type: int},
{name: b, type: int}
]}'
)
{code}
And we try to query it:
{code}
select * from tmp
{code}
and we get an exception
{code}
13/11/14 17:12:15 WARN avro.AvroSerdeUtils: Encountered AvroSerdeException
determining schema. Returning signal schema to indicate problem
org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Neither
avro.schema.literal nor avro.schema.url specified, can't determine table
schema
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:66)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:249)
at
org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:251)
at
org.apache.hadoop.hive.ql.metadata.Partition.initialize(Partition.java:217)
at
org.apache.hadoop.hive.ql.metadata.Partition.init(Partition.java:107)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:1573)
at
org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:190)
at
org.apache.hadoop.hive.ql.parse.ParseContext.getPrunedPartitions(ParseContext.java:561)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.checkTree(SimpleFetchOptimizer.java:144)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:74)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:102)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8200)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.checkedCompile(BeeswaxServiceImpl.java:247)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.compile(BeeswaxServiceImpl.java:200)
at
com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:830)
at
com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:823)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
[jira] [Updated] (HIVE-6827) Disable insecure commands with std sql auth
[ https://issues.apache.org/jira/browse/HIVE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6827: --- Status: Patch Available (was: Reopened) Disable insecure commands with std sql auth --- Key: HIVE-6827 URL: https://issues.apache.org/jira/browse/HIVE-6827 Project: Hive Issue Type: Task Components: Authorization, Security Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6827.2.patch, HIVE-6827.patch Disable insecure command on auth V2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HIVE-6827) Disable insecure commands with std sql auth
[ https://issues.apache.org/jira/browse/HIVE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reopened HIVE-6827: Disable insecure commands with std sql auth --- Key: HIVE-6827 URL: https://issues.apache.org/jira/browse/HIVE-6827 Project: Hive Issue Type: Task Components: Authorization, Security Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6827.2.patch, HIVE-6827.patch Disable insecure command on auth V2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6827) Disable insecure commands with std sql auth
[ https://issues.apache.org/jira/browse/HIVE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6827: --- Attachment: HIVE-6827.2.patch Disable insecure commands with std sql auth --- Key: HIVE-6827 URL: https://issues.apache.org/jira/browse/HIVE-6827 Project: Hive Issue Type: Task Components: Authorization, Security Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6827.2.patch, HIVE-6827.patch Disable insecure command on auth V2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960161#comment-13960161 ] Lefty Leverenz commented on HIVE-6109: -- Good catch, [~satish.mittal]. You get the eagle-eye award. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Fix For: 0.13.0 Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960186#comment-13960186 ] Ashutosh Chauhan commented on HIVE-6319: [~alangates] Can you update RB with your latest patch ? Insert, update, delete functionality needs a compactor -- Key: HIVE-6319 URL: https://issues.apache.org/jira/browse/HIVE-6319 Project: Hive Issue Type: Sub-task Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.13.0 Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf In order to keep the number of delta files from spiraling out of control we need a compactor to collect these delta files together, and eventually rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960200#comment-13960200 ] Alan Gates commented on HIVE-6319: -- [~ashutoshc] Done. Insert, update, delete functionality needs a compactor -- Key: HIVE-6319 URL: https://issues.apache.org/jira/browse/HIVE-6319 Project: Hive Issue Type: Sub-task Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.13.0 Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf In order to keep the number of delta files from spiraling out of control we need a compactor to collect these delta files together, and eventually rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6411) Support more generic way of using composite key for HBaseHandler
[
https://issues.apache.org/jira/browse/HIVE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960208#comment-13960208
]
Swarnim Kulkarni commented on HIVE-6411:
[~xuefuz][~navis] I was wondering if there is something that is still holding
us back on this JIRA or can we get this in?
Support more generic way of using composite key for HBaseHandler
Key: HIVE-6411
URL: https://issues.apache.org/jira/browse/HIVE-6411
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-6411.1.patch.txt, HIVE-6411.2.patch.txt,
HIVE-6411.3.patch.txt, HIVE-6411.4.patch.txt, HIVE-6411.5.patch.txt,
HIVE-6411.6.patch.txt, HIVE-6411.7.patch.txt, HIVE-6411.8.patch.txt,
HIVE-6411.9.patch.txt
HIVE-2599 introduced using custom object for the row key. But it forces key
objects to extend HBaseCompositeKey, which is again extension of LazyStruct.
If user provides proper Object and OI, we can replace internal key and keyOI
with those.
Initial implementation is based on factory interface.
{code}
public interface HBaseKeyFactory {
void init(SerDeParameters parameters, Properties properties) throws
SerDeException;
ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
LazyObjectBase createObject(ObjectInspector inspector) throws
SerDeException;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-4616) Simple reconnection support for jdbc2
[
https://issues.apache.org/jira/browse/HIVE-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960217#comment-13960217
]
Hive QA commented on HIVE-4616:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638425/HIVE-4616.3.patch.txt
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5546 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2113/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2113/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638425
Simple reconnection support for jdbc2
-
Key: HIVE-4616
URL: https://issues.apache.org/jira/browse/HIVE-4616
Project: Hive
Issue Type: Improvement
Components: JDBC
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-4616.3.patch.txt, HIVE-4616.D10953.1.patch,
HIVE-4616.D10953.2.patch
jdbc:hive2://localhost:1/db2;autoReconnect=true
simple reconnection on TransportException. If hiveserver2 has not been
shutdown, session could be reused.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6836) Upgrade parquet to 1.4.0
[ https://issues.apache.org/jira/browse/HIVE-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960241#comment-13960241 ] Szehon Ho commented on HIVE-6836: - Hi [~brocknoland], do you mind reviewing when you get the chance? Upgrade parquet to 1.4.0 Key: HIVE-6836 URL: https://issues.apache.org/jira/browse/HIVE-6836 Project: Hive Issue Type: Task Components: Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-6836.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6842) Hive should call input/output format methods in a more appropriate place
Sergey Shelukhin created HIVE-6842: -- Summary: Hive should call input/output format methods in a more appropriate place Key: HIVE-6842 URL: https://issues.apache.org/jira/browse/HIVE-6842 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin While investigating while HBase storage handler works on MR but not Tez (HIVE-6739), I've discovered that the crucial part on MR path that sets some jobconf properties is the call to MapWork::deriveExplainAttributes. It seems like this should be done in some more appropriate place. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6842) Hive should call input/output format methods in a more appropriate place
[ https://issues.apache.org/jira/browse/HIVE-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6842: --- Description: While investigating why HBase storage handler works on MR but not Tez (HIVE-6739), I've discovered that the crucial part on MR path that sets some jobconf properties is the call to MapWork::deriveExplainAttributes. It seems like this should be done in some more appropriate place. was: While investigating while HBase storage handler works on MR but not Tez (HIVE-6739), I've discovered that the crucial part on MR path that sets some jobconf properties is the call to MapWork::deriveExplainAttributes. It seems like this should be done in some more appropriate place. Hive should call input/output format methods in a more appropriate place Key: HIVE-6842 URL: https://issues.apache.org/jira/browse/HIVE-6842 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin While investigating why HBase storage handler works on MR but not Tez (HIVE-6739), I've discovered that the crucial part on MR path that sets some jobconf properties is the call to MapWork::deriveExplainAttributes. It seems like this should be done in some more appropriate place. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6056) The AvroSerDe gives out BadSchemaException if a partition is added to the table
[
https://issues.apache.org/jira/browse/HIVE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960243#comment-13960243
]
Ryan Tabora commented on HIVE-6056:
---
Is there any priority on this issue? This seems like a greatly debilitating bug.
The AvroSerDe gives out BadSchemaException if a partition is added to the
table
---
Key: HIVE-6056
URL: https://issues.apache.org/jira/browse/HIVE-6056
Project: Hive
Issue Type: Bug
Components: Database/Schema
Affects Versions: 0.11.0
Environment: amazon EMR (hadoop Amazon 1.0.3), avro-1.7.5
Reporter: Rushil Gupta
While creating an external table if I do not add a partition, I am able to
read files using following format:
{code}
CREATE external TABLE event
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 's3n://test-event/input/2013/14/10'
TBLPROPERTIES ('avro.schema.literal' = '..some schema..');
{code}
but if I add a partition based on date
{code}
CREATE external TABLE event
PARTITIONED BY (ds STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 's3n://test-event/input/'
TBLPROPERTIES ('avro.schema.literal' = '..some schema..');
ALTER TABLE event ADD IF NOT EXISTS PARTITION (ds = '2013_12_16') LOCATION
'2013/12/16/';
{code}
I get the following exception:
{code}
java.io.IOException:org.apache.hadoop.hive.serde2.avro.BadSchemaException
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6827) Disable insecure commands with std sql auth
[ https://issues.apache.org/jira/browse/HIVE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960263#comment-13960263 ] Thejas M Nair commented on HIVE-6827: - +1 I think we need to look at ways of allowing some set commands to be run, for example ones that are used to turn new optimizations on/off. I will create a followup jira for that. Disable insecure commands with std sql auth --- Key: HIVE-6827 URL: https://issues.apache.org/jira/browse/HIVE-6827 Project: Hive Issue Type: Task Components: Authorization, Security Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6827.2.patch, HIVE-6827.patch Disable insecure command on auth V2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6411) Support more generic way of using composite key for HBaseHandler
[
https://issues.apache.org/jira/browse/HIVE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960268#comment-13960268
]
Xuefu Zhang commented on HIVE-6411:
---
The only thing that's holding this is time: I haven't got a chance to go over
the latest patch yet. One thing that I'm unclear about is the predicate
pushdown related changes, per my previous comments.
{quote}
One thing that I don't quite get is the changes related to predicate pushdown.
Is this necessary?
{quote}
An explanation of the nature of the change should help. Otherwise, it may take
more of my time digging around.
Support more generic way of using composite key for HBaseHandler
Key: HIVE-6411
URL: https://issues.apache.org/jira/browse/HIVE-6411
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-6411.1.patch.txt, HIVE-6411.2.patch.txt,
HIVE-6411.3.patch.txt, HIVE-6411.4.patch.txt, HIVE-6411.5.patch.txt,
HIVE-6411.6.patch.txt, HIVE-6411.7.patch.txt, HIVE-6411.8.patch.txt,
HIVE-6411.9.patch.txt
HIVE-2599 introduced using custom object for the row key. But it forces key
objects to extend HBaseCompositeKey, which is again extension of LazyStruct.
If user provides proper Object and OI, we can replace internal key and keyOI
with those.
Initial implementation is based on factory interface.
{code}
public interface HBaseKeyFactory {
void init(SerDeParameters parameters, Properties properties) throws
SerDeException;
ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
LazyObjectBase createObject(ObjectInspector inspector) throws
SerDeException;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6821) Fix some non-deterministic tests
[ https://issues.apache.org/jira/browse/HIVE-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6821: - Status: Open (was: Patch Available) Looks like I missed a couple of .q.out files in patch, will regenerate patch. Not sure how TestCommandProcessorFactory.testAvailableCommands would be affected by this patch, since it's only modifying .q files. Fix some non-deterministic tests - Key: HIVE-6821 URL: https://issues.apache.org/jira/browse/HIVE-6821 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6821.1.patch A bunch of qfile tests look like they need an ORDER-BY added to the queries so that the output looks repeatable when testing with hadoop1/hadoop2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6739: --- Attachment: HIVE-6739.03.patch Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960279#comment-13960279 ] Vikram Dixit K commented on HIVE-6739: -- LGTM +1 Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6739: - Status: Open (was: Patch Available) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Status: Patch Available (was: Open) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960293#comment-13960293 ] Vikram Dixit K commented on HIVE-6739: -- Ignore previous comment. Wrong jira. Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6739: - Attachment: (was: HIVE-6782.6.patch) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Status: Open (was: Patch Available) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Attachment: HIVE-6782.6.patch HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6739: - Status: Patch Available (was: Open) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6739: - Attachment: HIVE-6782.6.patch Fix failing test. Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Attachment: HIVE-6782.6.patch HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6836) Upgrade parquet to 1.4.0
[ https://issues.apache.org/jira/browse/HIVE-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960295#comment-13960295 ] Brock Noland commented on HIVE-6836: +1 pending tests Upgrade parquet to 1.4.0 Key: HIVE-6836 URL: https://issues.apache.org/jira/browse/HIVE-6836 Project: Hive Issue Type: Task Components: Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-6836.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Attachment: (was: HIVE-6782.6.patch) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6817) Some hadoop2-only tests need diffs to be updated
[ https://issues.apache.org/jira/browse/HIVE-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960303#comment-13960303 ] Jason Dere commented on HIVE-6817: -- Looks like the diff for combine2.q has changed due to HIVE-6808, will need to regenerate the expected output. Also want to include TestHBaseNegativeCliDriver.cascade_dbdrop.q to this patch. Some hadoop2-only tests need diffs to be updated Key: HIVE-6817 URL: https://issues.apache.org/jira/browse/HIVE-6817 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6817.1.patch, HIVE-6817.2.patch expected output needs updating due to pre/post hook messages from the authorization changes -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6817) Some hadoop2-only tests need diffs to be updated
[ https://issues.apache.org/jira/browse/HIVE-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6817: - Attachment: HIVE-6817.2.patch patch v2 Some hadoop2-only tests need diffs to be updated Key: HIVE-6817 URL: https://issues.apache.org/jira/browse/HIVE-6817 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6817.1.patch, HIVE-6817.2.patch expected output needs updating due to pre/post hook messages from the authorization changes -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6817) Some hadoop2-only tests need diffs to be updated
[ https://issues.apache.org/jira/browse/HIVE-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960309#comment-13960309 ] Jason Dere commented on HIVE-6817: -- [~ashutoshc], does this still look ok? Some hadoop2-only tests need diffs to be updated Key: HIVE-6817 URL: https://issues.apache.org/jira/browse/HIVE-6817 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6817.1.patch, HIVE-6817.2.patch expected output needs updating due to pre/post hook messages from the authorization changes -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6411) Support more generic way of using composite key for HBaseHandler
[
https://issues.apache.org/jira/browse/HIVE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960308#comment-13960308
]
Swarnim Kulkarni commented on HIVE-6411:
I think part of that came from merging of my patch from
https://issues.apache.org/jira/browse/HIVE-6290 which added a support for
filter pushdown for structs.
Support more generic way of using composite key for HBaseHandler
Key: HIVE-6411
URL: https://issues.apache.org/jira/browse/HIVE-6411
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-6411.1.patch.txt, HIVE-6411.2.patch.txt,
HIVE-6411.3.patch.txt, HIVE-6411.4.patch.txt, HIVE-6411.5.patch.txt,
HIVE-6411.6.patch.txt, HIVE-6411.7.patch.txt, HIVE-6411.8.patch.txt,
HIVE-6411.9.patch.txt
HIVE-2599 introduced using custom object for the row key. But it forces key
objects to extend HBaseCompositeKey, which is again extension of LazyStruct.
If user provides proper Object and OI, we can replace internal key and keyOI
with those.
Initial implementation is based on factory interface.
{code}
public interface HBaseKeyFactory {
void init(SerDeParameters parameters, Properties properties) throws
SerDeException;
ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
LazyObjectBase createObject(ObjectInspector inspector) throws
SerDeException;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 19789: HIVE-6739 Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19789/ --- (Updated April 4, 2014, 7:55 p.m.) Review request for hive, Gunther Hagleitner and Vikram Dixit Kumaraswamy. Repository: hive-git Description --- See jira Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1ab90ca ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 720b8d5 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 5f0f353 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 385e22e ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fdbd996 ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 38c4c11 ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java e1cc3f4 ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java 996268e ql/src/java/org/apache/hadoop/hive/ql/plan/UnionWork.java 60781e6 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 93eccf8 ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java d2c332c ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 5ad4250 ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 4810fd0 Diff: https://reviews.apache.org/r/19789/diff/ Testing --- Thanks, Sergey Shelukhin
[jira] [Created] (HIVE-6843) INSTR for UTF-8 returns incorrect position
Clif Kranish created HIVE-6843: -- Summary: INSTR for UTF-8 returns incorrect position Key: HIVE-6843 URL: https://issues.apache.org/jira/browse/HIVE-6843 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.12.0, 0.11.0 Reporter: Clif Kranish Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960355#comment-13960355
]
Hive QA commented on HIVE-5799:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638427/HIVE-5799.8.patch.txt
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 5475 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
org.apache.hive.beeline.TestBeeLineWithArgs.org.apache.hive.beeline.TestBeeLineWithArgs
org.apache.hive.jdbc.TestJdbcDriver2.org.apache.hive.jdbc.TestJdbcDriver2
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnection
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnectionSchemaAPIs
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testURIDatabaseName
org.apache.hive.jdbc.TestJdbcWithMiniMr.org.apache.hive.jdbc.TestJdbcWithMiniMr
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithURL
org.apache.hive.jdbc.TestSSL.testSSLFetch
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.testAuthorization1
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testGetVariableValue
org.apache.hive.jdbc.miniHS2.TestMiniHS2.testConfInSession
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testConfOverlay
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatement
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testGetFunctions
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatement
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testGetFunctions
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testExecuteStatement
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testExecuteStatementAsync
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testGetFunctions
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2114/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2114/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638427
session/operation timeout for hiveserver2
-
Key: HIVE-5799
URL: https://issues.apache.org/jira/browse/HIVE-5799
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-5799.1.patch.txt, HIVE-5799.2.patch.txt,
HIVE-5799.3.patch.txt, HIVE-5799.4.patch.txt, HIVE-5799.5.patch.txt,
HIVE-5799.6.patch.txt, HIVE-5799.7.patch.txt, HIVE-5799.8.patch.txt
Need some timeout facility for preventing resource leakages from instable or
bad clients.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6843) INSTR for UTF-8 returns incorrect position
[ https://issues.apache.org/jira/browse/HIVE-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960354#comment-13960354 ] Clif Kranish commented on HIVE-6843: Using the INSTR fuction to find the posistion of a substring for a UTF-8 returns zero select INSTR (‘НАСТРОЕние’, ‘P’) from foo-bar INSTR for UTF-8 returns incorrect position -- Key: HIVE-6843 URL: https://issues.apache.org/jira/browse/HIVE-6843 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.11.0, 0.12.0 Reporter: Clif Kranish Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6824) Hive HBase query fails on Tez due to missing jars - part 2
[ https://issues.apache.org/jira/browse/HIVE-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960365#comment-13960365 ] Vikram Dixit K commented on HIVE-6824: -- +1 LGTM. Hive HBase query fails on Tez due to missing jars - part 2 -- Key: HIVE-6824 URL: https://issues.apache.org/jira/browse/HIVE-6824 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-6824.patch Follow-up from HIVE-6739. We cannot wait for Tez 0.4 (or even be sure that it will have TEZ-1004 and TEZ-1005), so I will split the patch into two. Original jira will have the straightforward (but less efficient) fix. This jira will use new relocalize APIs. Depending on relative timing of Tez 0.4 release and Hive 0.13 release, this will go into 0.13 or 0.14 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6841: --- Attachment: HIVE-6841.2.patch Updated patch maintains existing behavior for partitions on string columns. Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6841: --- Status: Patch Available (was: Open) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6841: --- Status: Open (was: Patch Available) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960379#comment-13960379 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-6841: - +1 The projection of (ctinyint=__HIVE_DEFAULT_PARTITION__) to NULL should be documented. Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongjie Chen updated HIVE-6784: --- Attachment: HIVE-6784.1.patch.txt parquet-hive should allow column type change Key: HIVE-6784 URL: https://issues.apache.org/jira/browse/HIVE-6784 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Tongjie Chen Attachments: HIVE-6784.1.patch.txt see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/323 Currently, if we change parquet format hive table using alter table parquet_table change c1 c1 bigint ( assuming original type of c1 is int), it will result in exception thrown from SerDe: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable in query runtime. This is different behavior from hive (using other file format), where it will try to perform cast (null value in case of incompatible type). -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19978: HIVE-6821 Fix some non-deterministic tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19978/ --- (Updated April 4, 2014, 8:59 p.m.) Review request for hive and Ashutosh Chauhan. Changes --- Generate diff using git diff -a. Also update diff for merge3.q due to HIVE-6808. Bugs: HIVE-6821 https://issues.apache.org/jira/browse/HIVE-6821 Repository: hive-git Description --- - Fix a bunch of qfile tests to have same output between hadoop1/hadoop2 by adding ORDER BY to queries - compile_processor.q should drop temp function to allow show_functions.q to pass when qfile tests run sequentially. Diffs (updated) - ql/src/test/queries/clientpositive/avro_partitioned.q 068a13c ql/src/test/queries/clientpositive/compile_processor.q 56e561c ql/src/test/queries/clientpositive/list_bucket_dml_4.q 918c817 ql/src/test/queries/clientpositive/load_dyn_part1.q 52b4937 ql/src/test/queries/clientpositive/load_dyn_part10.q 9517664 ql/src/test/queries/clientpositive/load_dyn_part3.q e4c8c17 ql/src/test/queries/clientpositive/load_dyn_part4.q 3f3a0c8 ql/src/test/queries/clientpositive/load_dyn_part8.q 8073500 ql/src/test/queries/clientpositive/load_dyn_part9.q 01fa596 ql/src/test/queries/clientpositive/merge3.q aacd0cd ql/src/test/queries/clientpositive/merge4.q 744783b ql/src/test/queries/clientpositive/mi.q 067c143 ql/src/test/queries/clientpositive/orc_create.q 9ab841a ql/src/test/queries/clientpositive/parquet_partitioned.q 103d26f ql/src/test/queries/clientpositive/stats4.q 6258004 ql/src/test/queries/clientpositive/subquery_in.q 37bb5d7 ql/src/test/queries/clientpositive/subquery_in_having.q 6dd55bf ql/src/test/queries/clientpositive/subquery_views.q e1e5373 ql/src/test/queries/clientpositive/union_top_level.q ff9d92c ql/src/test/queries/clientpositive/vectorization_15.q 35f03db ql/src/test/queries/clientpositive/vectorization_part_project.q 09c02b3 ql/src/test/queries/clientpositive/vectorized_rcfile_columnar.q 660442f ql/src/test/queries/clientpositive/vectorized_timestamp_funcs.q 6975839 ql/src/test/results/clientpositive/avro_partitioned.q.out 352ec0d ql/src/test/results/clientpositive/compile_processor.q.out b164663 ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 93f7dff ql/src/test/results/clientpositive/load_dyn_part1.q.out 787ebb9 ql/src/test/results/clientpositive/load_dyn_part10.q.out 90dd5d2 ql/src/test/results/clientpositive/load_dyn_part3.q.out 06d31d9 ql/src/test/results/clientpositive/load_dyn_part4.q.out 4064125 ql/src/test/results/clientpositive/load_dyn_part8.q.out 5f9ed27 ql/src/test/results/clientpositive/load_dyn_part9.q.out eb4183d ql/src/test/results/clientpositive/merge3.q.out 839ba66 ql/src/test/results/clientpositive/merge4.q.out dd53baa ql/src/test/results/clientpositive/mi.q.out 96410f7 ql/src/test/results/clientpositive/orc_create.q.out d15cf59 ql/src/test/results/clientpositive/parquet_partitioned.q.out 107702d ql/src/test/results/clientpositive/stats4.q.out 7662e16 ql/src/test/results/clientpositive/subquery_in.q.out f3e357d ql/src/test/results/clientpositive/subquery_in_having.q.out b7de423 ql/src/test/results/clientpositive/subquery_views.q.out f1fd6ee ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out ea0f1b9 ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out 75de746 ql/src/test/results/clientpositive/tez/subquery_in.q.out 201a890 ql/src/test/results/clientpositive/tez/vectorization_15.q.out 717a3cb ql/src/test/results/clientpositive/union_top_level.q.out 64352d4 ql/src/test/results/clientpositive/vectorization_15.q.out 717a3cb ql/src/test/results/clientpositive/vectorization_part_project.q.out c3121e5 ql/src/test/results/clientpositive/vectorized_rcfile_columnar.q.out b44c296 ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.out c964692 Diff: https://reviews.apache.org/r/19978/diff/ Testing --- Thanks, Jason Dere
[jira] [Updated] (HIVE-6821) Fix some non-deterministic tests
[ https://issues.apache.org/jira/browse/HIVE-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6821: - Attachment: HIVE-6821.2.patch patch v2 - generate diff using git diff -a to include the missing .q.out files (which git treated as binary files). Also regenerated merge3.q.out since it had changed due to HIVE-6808. Fix some non-deterministic tests - Key: HIVE-6821 URL: https://issues.apache.org/jira/browse/HIVE-6821 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6821.1.patch, HIVE-6821.2.patch A bunch of qfile tests look like they need an ORDER-BY added to the queries so that the output looks repeatable when testing with hadoop1/hadoop2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6821) Fix some non-deterministic tests
[ https://issues.apache.org/jira/browse/HIVE-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6821: - Status: Patch Available (was: Open) Fix some non-deterministic tests - Key: HIVE-6821 URL: https://issues.apache.org/jira/browse/HIVE-6821 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6821.1.patch, HIVE-6821.2.patch A bunch of qfile tests look like they need an ORDER-BY added to the queries so that the output looks repeatable when testing with hadoop1/hadoop2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960421#comment-13960421 ] Tongjie Chen commented on HIVE-6784: https://reviews.apache.org/r/20048/ parquet-hive should allow column type change Key: HIVE-6784 URL: https://issues.apache.org/jira/browse/HIVE-6784 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Tongjie Chen Attachments: HIVE-6784.1.patch.txt see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/323 Currently, if we change parquet format hive table using alter table parquet_table change c1 c1 bigint ( assuming original type of c1 is int), it will result in exception thrown from SerDe: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable in query runtime. This is different behavior from hive (using other file format), where it will try to perform cast (null value in case of incompatible type). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6844) support separate configuration param for enabling authorization using new interface
Thejas M Nair created HIVE-6844: --- Summary: support separate configuration param for enabling authorization using new interface Key: HIVE-6844 URL: https://issues.apache.org/jira/browse/HIVE-6844 Project: Hive Issue Type: Bug Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair The existing configuration parameter *hive.security.authorization.enabled* is used for both SQL query level authorization at sql query compilation, and at metatore api authorization for the thrift metastore api calls. This makes it hard to flexibly/correctly configure the security settings. It should be possible to enable SQL query level authorization and metastore api authorization independently of each other. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongjie Chen updated HIVE-6784: --- Description: see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/323 Currently, if we change parquet format hive table using alter table parquet_table change c1 c1 bigint ( assuming original type of c1 is int), it will result in exception thrown from SerDe: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable in query runtime. This is different behavior from hive (using other file format), where it will try to perform cast (null value in case of incompatible type). Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored in footers of parquet files); ParquetHiveSerDe also creates an corresponding ArrayWritableObjectInspector (but using column type info from metastore). Whenever there is column type change, the objector inspector will throw exception, since WritableLongObjectInspector cannot inspect an IntWritable etc... Conversion has to happen somewhere if we want to allow type change. SerDe's deserialize method seems a natural place for it. Currently, serialize method calls createStruct (then createPrimitive) for every record, but it creates a new object regardless, which seems expensive. I think that could be optimized a bit by just returning the object passed if already of the right type. deserialize also reuse this method, if there is a type change, there will be new object to be created, which I think is inevitable. was: see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/323 Currently, if we change parquet format hive table using alter table parquet_table change c1 c1 bigint ( assuming original type of c1 is int), it will result in exception thrown from SerDe: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable in query runtime. This is different behavior from hive (using other file format), where it will try to perform cast (null value in case of incompatible type). parquet-hive should allow column type change Key: HIVE-6784 URL: https://issues.apache.org/jira/browse/HIVE-6784 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Tongjie Chen Attachments: HIVE-6784.1.patch.txt see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/323 Currently, if we change parquet format hive table using alter table parquet_table change c1 c1 bigint ( assuming original type of c1 is int), it will result in exception thrown from SerDe: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable in query runtime. This is different behavior from hive (using other file format), where it will try to perform cast (null value in case of incompatible type). Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored in footers of parquet files); ParquetHiveSerDe also creates an corresponding ArrayWritableObjectInspector (but using column type info from metastore). Whenever there is column type change, the objector inspector will throw exception, since WritableLongObjectInspector cannot inspect an IntWritable etc... Conversion has to happen somewhere if we want to allow type change. SerDe's deserialize method seems a natural place for it. Currently, serialize method calls createStruct (then createPrimitive) for every record, but it creates a new object regardless, which seems expensive. I think that could be optimized a bit by just returning the object passed if already of the right type. deserialize also reuse this method, if there is a type change, there will be new object to be created, which I think is inevitable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6844) support separate configuration param for enabling authorization using new interface
[ https://issues.apache.org/jira/browse/HIVE-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960422#comment-13960422 ] Thejas M Nair commented on HIVE-6844: - Since the parameter has been used for both types of security, for backward compatibility reasons, if old authorization interface is used, it will still be enabled if hive.security.authorization.enabled=true. support separate configuration param for enabling authorization using new interface --- Key: HIVE-6844 URL: https://issues.apache.org/jira/browse/HIVE-6844 Project: Hive Issue Type: Bug Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair The existing configuration parameter *hive.security.authorization.enabled* is used for both SQL query level authorization at sql query compilation, and at metatore api authorization for the thrift metastore api calls. This makes it hard to flexibly/correctly configure the security settings. It should be possible to enable SQL query level authorization and metastore api authorization independently of each other. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960434#comment-13960434 ] Ashutosh Chauhan commented on HIVE-6841: +1 Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6817) Some hadoop2-only tests need diffs to be updated
[ https://issues.apache.org/jira/browse/HIVE-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960457#comment-13960457 ] Ashutosh Chauhan commented on HIVE-6817: yup.. LGTM Some hadoop2-only tests need diffs to be updated Key: HIVE-6817 URL: https://issues.apache.org/jira/browse/HIVE-6817 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6817.1.patch, HIVE-6817.2.patch expected output needs updating due to pre/post hook messages from the authorization changes -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4904) A little more CP crossing RS boundaries
[ https://issues.apache.org/jira/browse/HIVE-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-4904: Attachment: HIVE-4904.3.patch A little more CP crossing RS boundaries --- Key: HIVE-4904 URL: https://issues.apache.org/jira/browse/HIVE-4904 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4904.3.patch, HIVE-4904.D11757.1.patch, HIVE-4904.D11757.2.patch Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4904) A little more CP crossing RS boundaries
[ https://issues.apache.org/jira/browse/HIVE-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960465#comment-13960465 ] Harish Butani commented on HIVE-4904: - Rebased earlier patch. - Some of the changes to SemAly already went in, probably in HIVE-4293 - added couple of more checks: - When pruning the RedSink signature create a new ArrayList. The issue was the ArrayList in the RowSchema was being shared with the RowSchema of the ParentOp. - When there are distinct aggregations, the resolution back from GBy to RedSink outputCols is not working. For now I have added a check that adds all Value Cols of the RedSink to the prunedList of the RedSink Still need to add the test from HIVE-6912. A little more CP crossing RS boundaries --- Key: HIVE-4904 URL: https://issues.apache.org/jira/browse/HIVE-4904 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4904.3.patch, HIVE-4904.D11757.1.patch, HIVE-4904.D11757.2.patch Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 20051: HIVE-4904: A little more CP crossing RS boundaries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20051/ --- Review request for hive, Ashutosh Chauhan and Navis Ryu. Bugs: HIVE-4904 https://issues.apache.org/jira/browse/HIVE-4904 Repository: hive-git Description --- Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPruner.java 58a9b59 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcCtx.java db36151 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 0690fb7 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 3f16dc2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java 94224b3 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3b33dc2 ql/src/test/results/clientpositive/auto_join18.q.out b8677f4 ql/src/test/results/clientpositive/auto_join27.q.out a576190 ql/src/test/results/clientpositive/auto_join30.q.out 8709198 ql/src/test/results/clientpositive/auto_join31.q.out 1936e45 ql/src/test/results/clientpositive/auto_join32.q.out 05f53e6 ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 8882aac ql/src/test/results/clientpositive/count.q.out eb048b6 ql/src/test/results/clientpositive/groupby2_map.q.out 291f196 ql/src/test/results/clientpositive/groupby2_map_skew.q.out d005b6c ql/src/test/results/clientpositive/groupby3_map.q.out 1dfee08 ql/src/test/results/clientpositive/groupby3_map_skew.q.out 7af59bc ql/src/test/results/clientpositive/groupby_cube1.q.out 92d81f4 ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out b405978 ql/src/test/results/clientpositive/groupby_map_ppr.q.out 27eff75 ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out ad76252 ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out 51a70c4 ql/src/test/results/clientpositive/groupby_position.q.out 727bccb ql/src/test/results/clientpositive/groupby_rollup1.q.out 36bf966 ql/src/test/results/clientpositive/groupby_sort_11.q.out 8ee7571 ql/src/test/results/clientpositive/groupby_sort_8.q.out a27cfaa ql/src/test/results/clientpositive/join18.q.out 7975c79 ql/src/test/results/clientpositive/metadataonly1.q.out 917efdf ql/src/test/results/clientpositive/multi_insert_gby2.q.out ab758cb ql/src/test/results/clientpositive/multi_insert_lateral_view.q.out 35e70b4 ql/src/test/results/clientpositive/nullgroup.q.out 2ac7dea ql/src/test/results/clientpositive/nullgroup2.q.out cf31dc1 ql/src/test/results/clientpositive/nullgroup4.q.out feae138 ql/src/test/results/clientpositive/nullgroup4_multi_distinct.q.out 2ee357f ql/src/test/results/clientpositive/reduce_deduplicate_extended.q.out 9c6d14e ql/src/test/results/clientpositive/udf_count.q.out fb45708 ql/src/test/results/clientpositive/union11.q.out f226f35 ql/src/test/results/clientpositive/union14.q.out a6d349b ql/src/test/results/clientpositive/union15.q.out 88c9553 ql/src/test/results/clientpositive/union16.q.out 2bd8d5e ql/src/test/results/clientpositive/union2.q.out 0fac9d9 ql/src/test/results/clientpositive/union28.q.out 4252062 ql/src/test/results/clientpositive/union30.q.out 194b3b8 ql/src/test/results/clientpositive/union31.q.out 2f7031f ql/src/test/results/clientpositive/union5.q.out 0087393 ql/src/test/results/clientpositive/union7.q.out 3a2d88c ql/src/test/results/clientpositive/union9.q.out c6cc511 Diff: https://reviews.apache.org/r/20051/diff/ Testing --- Thanks, Harish Butani
[jira] [Resolved] (HIVE-6819) Correctness issue with Hive limit operator predicate push down
[
https://issues.apache.org/jira/browse/HIVE-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Butani resolved HIVE-6819.
-
Resolution: Duplicate
Correctness issue with Hive limit operator predicate push down
Key: HIVE-6819
URL: https://issues.apache.org/jira/browse/HIVE-6819
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Fix For: 0.13.0
Attachments: HIVE-6819.1.patch
Following query produces 0 rows with Predicate Push Down optimization turned
on; the same query produces 130 rows with predicate push down turned off.
{code:sql}
SELECT t2.c_int FROM (select key, value, c_float, c_int from t1 ORDER BY key,
value, c_float, c_int LIMIT 10) t1 JOIN t2 on t1.c_int=t2.c_int and
t1.c_float=t2.c_float WHERE t2.c_int=1;
{code}
I could reproduce this on Apache Trunk.
Haven't checked if previous releases have the same issue.
hive desc t1;
Query ID = jpullokkaran_20140401191515_36e441c6-074b-45ae-aff6-489e13a6f401
OK
key string
value string
c_int int
c_float float
c_boolean boolean
Time taken: 0.077 seconds, Fetched: 5 row(s)
hive select distinct key, value, c_float, c_int from t1;
OK
1 1 1.0 1
1 1 1.0 1
1 1 1.0 1
1 1 1.0 1
null nullNULLNULL
Time taken: 0.062 seconds, Fetched: 5 row(s)
hive desc t2;
Query ID = jpullokkaran_20140401191616_dfbd14bb-b5b8-4165-8d01-e9a61a7f1c33
OK
key string
value string
c_int int
c_float float
c_boolean boolean
Time taken: 0.062 seconds, Fetched: 5 row(s)
hive select distinct key, value, c_float, c_int from t2;
OK
1 1 1.0 1
1 1 1.0 1
1 1 1.0 1
1 1 1.0 1
2 2 2.0 2
null nullNULLNULL
Time taken: 4.698 seconds, Fetched: 6 row(s)
hive select t2.c_int from (select key, value, c_float, c_int from t1 order
by key,value,c_float,c_int limit 10)t1 join t2 on t1.c_int=t2.c_int and
t1.c_float=t2.c_float where t2.c_int=1;
MapredLocal task succeeded
OK
Time taken: 13.029 seconds
hive
hive select t2.c_int from (select key, value, c_float, c_int from t1 order
by key,value,c_float,c_int limit 10)t1 join t2 on t1.c_int=t2.c_int and
t1.c_float=t2.c_float where t2.c_int=1;
MapredLocal task succeeded
OK
...
1
1
1
1
1
1
1
1
1
1
1
Time taken: 9.317 seconds, Fetched: 130 row(s)
hive
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HIVE-4904) A little more CP crossing RS boundaries
[ https://issues.apache.org/jira/browse/HIVE-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960465#comment-13960465 ] Harish Butani edited comment on HIVE-4904 at 4/4/14 9:57 PM: - Rebased earlier patch. - Some of the changes to SemAly already went in, probably in HIVE-4293 - added couple of more checks: - When pruning the RedSink signature create a new ArrayList. The issue was the ArrayList in the RowSchema was being shared with the RowSchema of the ParentOp. - When there are distinct aggregations, the resolution back from GBy to RedSink outputCols is not working. For now I have added a check that adds all Value Cols of the RedSink to the prunedList of the RedSink Still need to add the test from HIVE-6819 was (Author: rhbutani): Rebased earlier patch. - Some of the changes to SemAly already went in, probably in HIVE-4293 - added couple of more checks: - When pruning the RedSink signature create a new ArrayList. The issue was the ArrayList in the RowSchema was being shared with the RowSchema of the ParentOp. - When there are distinct aggregations, the resolution back from GBy to RedSink outputCols is not working. For now I have added a check that adds all Value Cols of the RedSink to the prunedList of the RedSink Still need to add the test from HIVE-6912. A little more CP crossing RS boundaries --- Key: HIVE-4904 URL: https://issues.apache.org/jira/browse/HIVE-4904 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4904.3.patch, HIVE-4904.D11757.1.patch, HIVE-4904.D11757.2.patch Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6829) alter table foo compact gives an error
[ https://issues.apache.org/jira/browse/HIVE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960478#comment-13960478 ] Harish Butani commented on HIVE-6829: - +1 for 0.13 alter table foo compact gives an error --- Key: HIVE-6829 URL: https://issues.apache.org/jira/browse/HIVE-6829 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-6829.patch Doing alter table foo compact 'minor' results in the error: ERROR ql.Driver (SessionState.java:printError(550)) - FAILED: HiveException Operation should not be null org.apache.hadoop.hive.ql.metadata.HiveException: Operation should not be null alter table foo partition (key='value') compact 'minor' works fine. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[
https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960491#comment-13960491
]
Hive QA commented on HIVE-6782:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638746/HIVE-6782.6.patch
{color:green}SUCCESS:{color} +1 5546 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2116/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2116/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638746
HiveServer2Concurrency issue when running with tez intermittently, throwing
org.apache.tez.dag.api.SessionNotRunning: Application not running error
-
Key: HIVE-6782
URL: https://issues.apache.org/jira/browse/HIVE-6782
Project: Hive
Issue Type: Bug
Components: Tez
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Fix For: 0.13.0, 0.14.0
Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch,
HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch
HiveServer2 concurrency is failing intermittently when using tez, throwing
org.apache.tez.dag.api.SessionNotRunning: Application not running error
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6845) TestJdbcDriver.testShowRoleGrant can fail if TestJdbcDriver/TestJdbcDriver2 run together
Jason Dere created HIVE-6845:
Summary: TestJdbcDriver.testShowRoleGrant can fail if
TestJdbcDriver/TestJdbcDriver2 run together
Key: HIVE-6845
URL: https://issues.apache.org/jira/browse/HIVE-6845
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Running both TestJdbcDriver/TestJdbcDriver2 together in the same run gives an
error in testShowRoleGrant() because both tests create the role role1. When
the 2nd test tries to create the role it fails:
{noformat}
testShowRoleGrant(org.apache.hive.jdbc.TestJdbcDriver2) Time elapsed: 1.801
sec ERROR!
java.sql.SQLException: Error while processing statement: FAILED: Execution
Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
at
org.apache.hive.jdbc.TestJdbcDriver2.testShowRoleGrant(TestJdbcDriver2.java:2000)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
