[jira] [Commented] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860083#comment-13860083 ] Thejas M Nair commented on HIVE-5923: - rb link https://reviews.apache.org/r/16565/ SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch, HIVE-5923.2.patch, HIVE-5923.3.patch Original Estimate: 96h Time Spent: 72h Remaining Estimate: 12h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5941) SQL std auth - support 'show all roles'
[
https://issues.apache.org/jira/browse/HIVE-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860265#comment-13860265
]
Brock Noland commented on HIVE-5941:
{noformat}
((FSDataOutputStream) outStream).close();
{noformat}
?
SQL std auth - support 'show all roles'
---
Key: HIVE-5941
URL: https://issues.apache.org/jira/browse/HIVE-5941
Project: Hive
Issue Type: Sub-task
Components: Authorization
Reporter: Thejas M Nair
Assignee: Navis
Attachments: HIVE-5941.1.patch.txt, HIVE-5941.2.patch.txt
Original Estimate: 24h
Remaining Estimate: 24h
SHOW ALL ROLES - This will list all
currently existing roles. This will be available only to the superuser.
This task includes parser changes.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Hive-trunk-hadoop2 - Build # 642 - Still Failing
Changes for Build #640 Changes for Build #641 [navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis reviewed by Thejas M Nair) [navis] HIVE-4257 : java.sql.SQLNonTransientConnectionException on JDBCStatsAggregator (Teddy Choi via Navis, reviewed by Ashutosh) Changes for Build #642 No tests ran. The Apache Jenkins build system has built Hive-trunk-hadoop2 (build #642) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-hadoop2/642/ to view the results.
Hive-trunk-h0.21 - Build # 2541 - Still Failing
Changes for Build #2539 Changes for Build #2540 [navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis reviewed by Thejas M Nair) Changes for Build #2541 No tests ran. The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2541) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2541/ to view the results.
Re: Review Request 15873: Query cancel should stop running MR tasks
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15873/#review31039 --- ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/15873/#comment59384 wouldn't this be true in case of many error paths above? Where driverCxt.shutdown(); is called - Sergey Shelukhin On Dec. 30, 2013, 6:57 a.m., Navis Ryu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15873/ --- (Updated Dec. 30, 2013, 6:57 a.m.) Review request for hive. Bugs: HIVE-5901 https://issues.apache.org/jira/browse/HIVE-5901 Repository: hive-git Description --- Currently, query canceling does not stop running MR job immediately. Diffs - ql/src/java/org/apache/hadoop/hive/ql/Driver.java 62fc150 ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 1c84523 ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java ead7b59 Diff: https://reviews.apache.org/r/15873/diff/ Testing --- Thanks, Navis Ryu
[jira] [Updated] (HIVE-5946) DDL authorization task factory should be better tested
[ https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5946: --- Summary: DDL authorization task factory should be better tested (was: DDL authorization task factory should be pluggable and tested) DDL authorization task factory should be better tested -- Key: HIVE-5946 URL: https://issues.apache.org/jira/browse/HIVE-5946 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5946.patch Thejas is working on various authorization issues and one element that might be useful in that effort and increase test coverage and testability would be perform authorization task creation in a factory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860380#comment-13860380 ] Sergey Shelukhin commented on HIVE-5757: This patch doesn't do what jira description lists. Also small nit on RB Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch Implement full, end-to-end support for CASE in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5946) DDL authorization task factory should be better tested
[ https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860385#comment-13860385 ] Brock Noland commented on HIVE-5946: [~thejas] I created this patch a while back after I noticed that the DDL code is monolithic and poorly tested. Since you are now making changes to this code path, it might make sense to commit this. Additionally I make the code path pluggable but I am now rethinking that aspect. If the pluggable aspect doesn't make sense then I can remove that and rebase the patch. DDL authorization task factory should be better tested -- Key: HIVE-5946 URL: https://issues.apache.org/jira/browse/HIVE-5946 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5946.patch Thejas is working on various authorization issues and one element that might be useful in that effort and increase test coverage and testability would be perform authorization task creation in a factory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5757: -- Description: Implement support for CASE in vectorized mode. The approach is to use the vectorized UDF adaptor internally. A higher-performance version that used VectorExpression subclasses was considered but not done due to complexity. Such a version potentially could be done in the future if it's important enough. This is high priority because CASE is a fairly popular expression. was: Implement full, end-to-end support for CASE in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because CASE is a fairly popular expression. Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch Implement support for CASE in vectorized mode. The approach is to use the vectorized UDF adaptor internally. A higher-performance version that used VectorExpression subclasses was considered but not done due to complexity. Such a version potentially could be done in the future if it's important enough. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860396#comment-13860396 ] Eric Hanson commented on HIVE-5757: --- Updated description to match the patch (saying that it uses the vectorized UDF adaptor internally). Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch Implement support for CASE in vectorized mode. The approach is to use the vectorized UDF adaptor internally. A higher-performance version that used VectorExpression subclasses was considered but not done due to complexity. Such a version potentially could be done in the future if it's important enough. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5958) SQL std auth - disable statements that work with paths
[ https://issues.apache.org/jira/browse/HIVE-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860411#comment-13860411 ] Alan Gates commented on HIVE-5958: -- As an intermediate could we allow URI operations (like LOAD and LOCATION for tables) only if the user executing the command owns the file or directory being referenced? This avoids the need to add full URI permissions into the first phase (thus making it harder to deliver some useful functionality) but delivers something that is hopefully still useful. SQL std auth - disable statements that work with paths -- Key: HIVE-5958 URL: https://issues.apache.org/jira/browse/HIVE-5958 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Original Estimate: 24h Remaining Estimate: 24h In the first pass, statement such as create table, alter table that specify an path uri will get an authorization error under SQL std auth . -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Review Request 16565: HIVE-5923 - SQL std auth - parser changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16565/#review31042 --- LGTM. Is this ready for commit? Also, can you check out HIVE-5946? I'd be happy to update that after committing this or do a follow on after committing this. It'd be great to unit test the DDLSemanticAnalyzer code and that patch adds about 20 tests. ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/16565/#comment59389 Thanks for cleaning this up.. - Brock Noland On Jan. 2, 2014, 7:05 a.m., Thejas Nair wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16565/ --- (Updated Jan. 2, 2014, 7:05 a.m.) Review request for hive and Brock Noland. Repository: hive-git Description --- Grant privilege and revoke privilege statements need to be changed to remove the requirement (but not the option) for the noise word TABLE. In the SQL specification table is the assumed default for grant and revoke statements. Today Hive’s syntax is GRANT action ON TABLE table TO grantee. It should be GRANT action ON [TABLE] table TO grantee. Grant role and revoke role statements has been changed to remove the need for keyword ROLE. Support for WITH ADMIN OPTION needs to be added to grant role and revoke role statement syntax. HIVE-5923.3.patch Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 8d0b7a8 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java a1b19a6 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 9354637 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g b34f53b ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 39d53d3 ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java f0acb16 ql/src/test/queries/clientpositive/authorization_2.q 4fc79b9 ql/src/test/queries/clientpositive/authorization_role_grant1.q PRE-CREATION ql/src/test/results/clientpositive/authorization_2.q.out 460240b ql/src/test/results/clientpositive/authorization_role_grant1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/16565/diff/ Testing --- Thanks, Thejas Nair
[jira] [Commented] (HIVE-5941) SQL std auth - support 'show all roles'
[ https://issues.apache.org/jira/browse/HIVE-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860417#comment-13860417 ] Brock Noland commented on HIVE-5941: Looks like that weird cast comes from code above. SQL std auth - support 'show all roles' --- Key: HIVE-5941 URL: https://issues.apache.org/jira/browse/HIVE-5941 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Navis Attachments: HIVE-5941.1.patch.txt, HIVE-5941.2.patch.txt Original Estimate: 24h Remaining Estimate: 24h SHOW ALL ROLES - This will list all currently existing roles. This will be available only to the superuser. This task includes parser changes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860421#comment-13860421 ] Eric Hanson commented on HIVE-5757: --- Updated patch per Sergey's feedback on RB. Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch, HIVE-5757.4.patch Implement support for CASE in vectorized mode. The approach is to use the vectorized UDF adaptor internally. A higher-performance version that used VectorExpression subclasses was considered but not done due to complexity. Such a version potentially could be done in the future if it's important enough. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5757: -- Attachment: HIVE-5757.4.patch Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch, HIVE-5757.4.patch Implement support for CASE in vectorized mode. The approach is to use the vectorized UDF adaptor internally. A higher-performance version that used VectorExpression subclasses was considered but not done due to complexity. Such a version potentially could be done in the future if it's important enough. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860497#comment-13860497 ] Jitendra Nath Pandey commented on HIVE-5757: +1, looks ok to me as well. Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch, HIVE-5757.4.patch Implement support for CASE in vectorized mode. The approach is to use the vectorized UDF adaptor internally. A higher-performance version that used VectorExpression subclasses was considered but not done due to complexity. Such a version potentially could be done in the future if it's important enough. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5757) Implement vectorized support for CASE
[ https://issues.apache.org/jira/browse/HIVE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860491#comment-13860491 ] Sergey Shelukhin commented on HIVE-5757: lgtm Implement vectorized support for CASE - Key: HIVE-5757 URL: https://issues.apache.org/jira/browse/HIVE-5757 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5757.1.patch, HIVE-5757.2.patch, HIVE-5757.3.patch, HIVE-5757.4.patch Implement support for CASE in vectorized mode. The approach is to use the vectorized UDF adaptor internally. A higher-performance version that used VectorExpression subclasses was considered but not done due to complexity. Such a version potentially could be done in the future if it's important enough. This is high priority because CASE is a fairly popular expression. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Switch Precommit builds to Java 7
Java 6 has been EOL for almost a year now and 7 is EOL in a short year and a quarter[1]. Therefore, I think it's time to switch the Precommit builds to Java 7. Brock 1. http://www.oracle.com/technetwork/java/eol-135779.html
[jira] [Commented] (HIVE-6105) LongWritable.compareTo needs shimming
[ https://issues.apache.org/jira/browse/HIVE-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860494#comment-13860494 ] Gunther Hagleitner commented on HIVE-6105: -- LGTM +1, will commit once tests have run. LongWritable.compareTo needs shimming - Key: HIVE-6105 URL: https://issues.apache.org/jira/browse/HIVE-6105 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Gunther Hagleitner Assignee: Navis Attachments: HIVE-6105.1.patch.txt Compiled hive against hadoop-2. Running tests on hadoop 1 gives: Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.io.LongWritable.compareTo(Lorg/apache/hadoop/io/LongWritable;)I at org.apache.hadoop.hive.ql.udf.UDAFPercentile$MyComparator.compare(UDAFPercentile.java:62) at org.apache.hadoop.hive.ql.udf.UDAFPercentile$MyComparator.compare(UDAFPercentile.java:58) at java.util.Arrays.mergeSort(Arrays.java:1270) at java.util.Arrays.sort(Arrays.java:1210) at java.util.Collections.sort(Collections.java:157) at org.apache.hadoop.hive.ql.udf.UDAFPercentile$PercentileLongEvaluator.terminate(UDAFPercentile.java:194) ... 20 more I think the problem is that the compareTo function has changed from using Object to using LongWritable as an argument. Seems this needs a shim. However, we should also make sure that this shim doesn't hit performance too hard. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6125) Tez: Refactoring changes
[ https://issues.apache.org/jira/browse/HIVE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860565#comment-13860565 ] Gunther Hagleitner commented on HIVE-6125: -- Thanks for the feedback [~leftylev]. I'll create new jira for that. Definitely useful to have them in hive-default.xml as another means of documentation. Tez: Refactoring changes Key: HIVE-6125 URL: https://issues.apache.org/jira/browse/HIVE-6125 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6125.1.patch In order to facilitate merge back I've separated out all the changes that don't require Tez. These changes introduce new interfaces, move code etc. In preparation of the Tez specific classes. This should help show what changes have been made that affect the MR codepath as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6128) Add tez variables to hive-default.xml
Gunther Hagleitner created HIVE-6128: Summary: Add tez variables to hive-default.xml Key: HIVE-6128 URL: https://issues.apache.org/jira/browse/HIVE-6128 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch hive.orc.splits.include.file.footer hive.orc.cache.stripe.details.size -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6128) Add tez variables to hive-default.xml
[ https://issues.apache.org/jira/browse/HIVE-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860568#comment-13860568 ] Gunther Hagleitner commented on HIVE-6128: -- + hive.orc.compute.splits.num.threads Add tez variables to hive-default.xml - Key: HIVE-6128 URL: https://issues.apache.org/jira/browse/HIVE-6128 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch hive.orc.splits.include.file.footer hive.orc.cache.stripe.details.size -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6125) Tez: Refactoring changes
[ https://issues.apache.org/jira/browse/HIVE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860569#comment-13860569 ] Gunther Hagleitner commented on HIVE-6125: -- Captured in HIVE-6128 Tez: Refactoring changes Key: HIVE-6125 URL: https://issues.apache.org/jira/browse/HIVE-6125 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-6125.1.patch In order to facilitate merge back I've separated out all the changes that don't require Tez. These changes introduce new interfaces, move code etc. In preparation of the Tez specific classes. This should help show what changes have been made that affect the MR codepath as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Review Request 16565: HIVE-5923 - SQL std auth - parser changes
On Jan. 2, 2014, 6:02 p.m., Brock Noland wrote: LGTM. Is this ready for commit? Also, can you check out HIVE-5946? I'd be happy to update that after committing this or do a follow on after committing this. It'd be great to unit test the DDLSemanticAnalyzer code and that patch adds about 20 tests. Yes, the patch is ready for review/commit. It needs a +1 and the full unit test suite to finish. I ran the auth tests locally and they pass. - Thejas --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16565/#review31042 --- On Jan. 2, 2014, 7:05 a.m., Thejas Nair wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16565/ --- (Updated Jan. 2, 2014, 7:05 a.m.) Review request for hive and Brock Noland. Repository: hive-git Description --- Grant privilege and revoke privilege statements need to be changed to remove the requirement (but not the option) for the noise word TABLE. In the SQL specification table is the assumed default for grant and revoke statements. Today Hive’s syntax is GRANT action ON TABLE table TO grantee. It should be GRANT action ON [TABLE] table TO grantee. Grant role and revoke role statements has been changed to remove the need for keyword ROLE. Support for WITH ADMIN OPTION needs to be added to grant role and revoke role statement syntax. HIVE-5923.3.patch Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 8d0b7a8 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java a1b19a6 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 9354637 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g b34f53b ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 39d53d3 ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java f0acb16 ql/src/test/queries/clientpositive/authorization_2.q 4fc79b9 ql/src/test/queries/clientpositive/authorization_role_grant1.q PRE-CREATION ql/src/test/results/clientpositive/authorization_2.q.out 460240b ql/src/test/results/clientpositive/authorization_role_grant1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/16565/diff/ Testing --- Thanks, Thejas Nair
Re: Review Request 16565: HIVE-5923 - SQL std auth - parser changes
On Jan. 2, 2014, 6:02 p.m., Brock Noland wrote: LGTM. Is this ready for commit? Also, can you check out HIVE-5946? I'd be happy to update that after committing this or do a follow on after committing this. It'd be great to unit test the DDLSemanticAnalyzer code and that patch adds about 20 tests. Thejas Nair wrote: Yes, the patch is ready for review/commit. It needs a +1 and the full unit test suite to finish. I ran the auth tests locally and they pass. I am reviewing HIVE-5946, the changes make it much easier to unit test ! - Thejas --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16565/#review31042 --- On Jan. 2, 2014, 7:05 a.m., Thejas Nair wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16565/ --- (Updated Jan. 2, 2014, 7:05 a.m.) Review request for hive and Brock Noland. Repository: hive-git Description --- Grant privilege and revoke privilege statements need to be changed to remove the requirement (but not the option) for the noise word TABLE. In the SQL specification table is the assumed default for grant and revoke statements. Today Hive’s syntax is GRANT action ON TABLE table TO grantee. It should be GRANT action ON [TABLE] table TO grantee. Grant role and revoke role statements has been changed to remove the need for keyword ROLE. Support for WITH ADMIN OPTION needs to be added to grant role and revoke role statement syntax. HIVE-5923.3.patch Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 8d0b7a8 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java a1b19a6 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 9354637 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g b34f53b ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 39d53d3 ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java f0acb16 ql/src/test/queries/clientpositive/authorization_2.q 4fc79b9 ql/src/test/queries/clientpositive/authorization_role_grant1.q PRE-CREATION ql/src/test/results/clientpositive/authorization_2.q.out 460240b ql/src/test/results/clientpositive/authorization_role_grant1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/16565/diff/ Testing --- Thanks, Thejas Nair
Precommit tests
Are pending due to: https://issues.apache.org/jira/browse/HADOOP-10199
[jira] [Updated] (HIVE-3611) Hive JDBC doesn't support BINARY column
[ https://issues.apache.org/jira/browse/HIVE-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-3611: -- Attachment: HIVE-3611.2.patch Patch #2 is based on patch #1, but with changes applied to Hive jdbc2. It's rebased with latest trunk. It also includes code cleanup and minor fixes. Hive JDBC doesn't support BINARY column --- Key: HIVE-3611 URL: https://issues.apache.org/jira/browse/HIVE-3611 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.9.0 Reporter: Viji Assignee: Xuefu Zhang Labels: patch Attachments: HIVE-3611.1.patch.txt, HIVE-3611.2.patch Similar to HIVE-2957, the JDBC HiveDriver should also support BINARY type columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-4216) TestHBaseMinimrCliDriver throws weird error with HBase 0.94.5 and Hadoop 23 and test is stuck infinitely
[
https://issues.apache.org/jira/browse/HIVE-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-4216:
-
Attachment: HIVE-4216.2.patch
New patch to fix failure I was seeing in TestHBaseCliDriver. Also fixing a
diff in TestHBaseNegativeCliDriver with -Phadoop-2.
TestHBaseMinimrCliDriver throws weird error with HBase 0.94.5 and Hadoop 23
and test is stuck infinitely
Key: HIVE-4216
URL: https://issues.apache.org/jira/browse/HIVE-4216
Project: Hive
Issue Type: Bug
Components: StorageHandler
Affects Versions: 0.9.0, 0.11.0, 0.12.0
Environment: Hadoop 23.X
Reporter: Viraj Bhat
Fix For: 0.13.0
Attachments: HIVE-4216.1.patch, HIVE-4216.2.patch
After upgrading to Hadoop 23 and HBase 0.94.5 compiled for Hadoop 23. The
TestHBaseMinimrCliDriver, fails after performing the following steps
Update hbase_bulk.m with the following properties
set mapreduce.totalorderpartitioner.naturalorder=false;
set mapreduce.totalorderpartitioner.path=/tmp/hbpartition.lst;
Otherwise I keep seeing: _partition.lst not found exception in the mappers,
even though set total.order.partitioner.path=/tmp/hbpartition.lst is set.
When the test runs, the 3 reducer phase of the second query fails with the
following error, but the MiniMRCluster keeps spinning up new reducer and the
test is stuck infinitely.
{code}
insert overwrite table hbsort
select distinct value,
case when key=103 then cast(null as string) else key end,
case when key=103 then ''
else cast(key+1 as string) end
from src
cluster by value;
{code}
The stack trace I see in the syslog for the Node Manager is the following:
==
13-03-20 16:26:48,942 FATAL [IPC Server handler 17 on 55996]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1363821864968_0003_r_02_0 - exited : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{key:{reducesinkkey0:val_200},value:{_col0:val_200,_col1:200,_col2:201.0},alias:0}
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:448)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0)
{key:{reducesinkkey0:val_200},value:{_col0:val_200,_col1:200,_col2:201.0},alias:0}
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
... 7 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
... 7 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.TaskID$CharTaskTypeMaps.getRepresentingCharacter(TaskID.java:265)
at org.apache.hadoop.mapreduce.TaskID.appendTo(TaskID.java:153)
at
org.apache.hadoop.mapreduce.TaskAttemptID.appendTo(TaskAttemptID.java:119)
at
org.apache.hadoop.mapreduce.TaskAttemptID.toString(TaskAttemptID.java:151)
at java.lang.String.valueOf(String.java:2826)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.init(FileOutputCommitter.java:69)
at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.getRecordWriter(HFileOutputFormat.java:90)
at
[jira] [Updated] (HIVE-4216) TestHBaseMinimrCliDriver throws weird error with HBase 0.94.5 and Hadoop 23 and test is stuck infinitely
[
https://issues.apache.org/jira/browse/HIVE-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-4216:
-
Status: Patch Available (was: Open)
TestHBaseMinimrCliDriver throws weird error with HBase 0.94.5 and Hadoop 23
and test is stuck infinitely
Key: HIVE-4216
URL: https://issues.apache.org/jira/browse/HIVE-4216
Project: Hive
Issue Type: Bug
Components: StorageHandler
Affects Versions: 0.12.0, 0.11.0, 0.9.0
Environment: Hadoop 23.X
Reporter: Viraj Bhat
Fix For: 0.13.0
Attachments: HIVE-4216.1.patch, HIVE-4216.2.patch
After upgrading to Hadoop 23 and HBase 0.94.5 compiled for Hadoop 23. The
TestHBaseMinimrCliDriver, fails after performing the following steps
Update hbase_bulk.m with the following properties
set mapreduce.totalorderpartitioner.naturalorder=false;
set mapreduce.totalorderpartitioner.path=/tmp/hbpartition.lst;
Otherwise I keep seeing: _partition.lst not found exception in the mappers,
even though set total.order.partitioner.path=/tmp/hbpartition.lst is set.
When the test runs, the 3 reducer phase of the second query fails with the
following error, but the MiniMRCluster keeps spinning up new reducer and the
test is stuck infinitely.
{code}
insert overwrite table hbsort
select distinct value,
case when key=103 then cast(null as string) else key end,
case when key=103 then ''
else cast(key+1 as string) end
from src
cluster by value;
{code}
The stack trace I see in the syslog for the Node Manager is the following:
==
13-03-20 16:26:48,942 FATAL [IPC Server handler 17 on 55996]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1363821864968_0003_r_02_0 - exited : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{key:{reducesinkkey0:val_200},value:{_col0:val_200,_col1:200,_col2:201.0},alias:0}
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:448)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0)
{key:{reducesinkkey0:val_200},value:{_col0:val_200,_col1:200,_col2:201.0},alias:0}
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
... 7 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
... 7 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.TaskID$CharTaskTypeMaps.getRepresentingCharacter(TaskID.java:265)
at org.apache.hadoop.mapreduce.TaskID.appendTo(TaskID.java:153)
at
org.apache.hadoop.mapreduce.TaskAttemptID.appendTo(TaskAttemptID.java:119)
at
org.apache.hadoop.mapreduce.TaskAttemptID.toString(TaskAttemptID.java:151)
at java.lang.String.valueOf(String.java:2826)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.init(FileOutputCommitter.java:69)
at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.getRecordWriter(HFileOutputFormat.java:90)
at
org.apache.hadoop.hive.hbase.HiveHFileOutputFormat.getFileWriter(HiveHFileOutputFormat.java:67)
at
Re: Review Request 16007: HIVE-5946 - DDL authorization task factory should be pluggable and tested
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16007/#review31067 --- ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestDefaultHiveAuthorizationTaskFactory.java https://reviews.apache.org/r/16007/#comment59407 A nit - I found it a little unintuitive to return the list from assertSize() call. Should we just have return type as void and do the check on another line. I don't have a strong opinion about this, so keeping it this way is also fine. - Thejas Nair On Dec. 4, 2013, 7:05 p.m., Brock Noland wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16007/ --- (Updated Dec. 4, 2013, 7:05 p.m.) Review request for hive. Bugs: HIVE-5946 https://issues.apache.org/jira/browse/HIVE-5946 Repository: hive-git Description --- Moves the task creation logic to a factory which is now tested with 20 unit tests. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36503fa911161f5bca1b50182481a5a6ee38ae42 ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d0807d23de740ede9abe054aab946f043d6918b0 ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java da80d817bb6e97db1d9a12e96a5c736ab2c3222b ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7443ea45cb1502090c1f6d03ce0c3b21d0bcaba4 ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/AbstractHiveAuthorizationTaskFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/DefaultHiveAuthorizationTaskFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryFactory.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestDefaultHiveAuthorizationTaskFactory.java PRE-CREATION Diff: https://reviews.apache.org/r/16007/diff/ Testing --- 20 additional unit tests which pass Thanks, Brock Noland
[jira] [Commented] (HIVE-5946) DDL authorization task factory should be better tested
[ https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860830#comment-13860830 ] Thejas M Nair commented on HIVE-5946: - Thanks Brock, this is very useful change, and it will help in adding more tests. I reviewed the changes and they look good to me. It also sets a good example for writing testable code! As you mention, I don't think we should make this pluggable. Once we make it 'pluggable' it would be considered a public API and we would need to start worrying about backward compatibility for it. DDL authorization task factory should be better tested -- Key: HIVE-5946 URL: https://issues.apache.org/jira/browse/HIVE-5946 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5946.patch Thejas is working on various authorization issues and one element that might be useful in that effort and increase test coverage and testability would be perform authorization task creation in a factory. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5941) SQL std auth - support 'show all roles'
[
https://issues.apache.org/jira/browse/HIVE-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860856#comment-13860856
]
Thejas M Nair commented on HIVE-5941:
-
{code}
outStream.writeBytes(role name: + roleName);
{code}
I think we should skip the role name: part in output. Can you also please
create a reviewboard link ?
SQL std auth - support 'show all roles'
---
Key: HIVE-5941
URL: https://issues.apache.org/jira/browse/HIVE-5941
Project: Hive
Issue Type: Sub-task
Components: Authorization
Reporter: Thejas M Nair
Assignee: Navis
Attachments: HIVE-5941.1.patch.txt, HIVE-5941.2.patch.txt
Original Estimate: 24h
Remaining Estimate: 24h
SHOW ALL ROLES - This will list all
currently existing roles. This will be available only to the superuser.
This task includes parser changes.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive
[ https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860859#comment-13860859 ] Eric Hanson commented on HIVE-6017: --- I checked up on this. Microsoft has signed the Apache Corporate Contributor License Agreement (CCLA) and the people who wrote the code signed the Apache ICLA. These agreements plus the copyright notice in the code cover the situation. Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive --- Key: HIVE-6017 URL: https://issues.apache.org/jira/browse/HIVE-6017 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-6017.01.patch, HIVE-6017.02.patch, HIVE-6017.03.patch, HIVE-6017.04.patch Contribute the Decimal128 high-performance decimal package developed by Microsoft to Hive. This was originally written for Microsoft PolyBase by Hideaki Kimura. This code is about 8X more efficient than Java BigDecimal for typical operations. It uses a finite (128 bit) precision and can handle up to decimal(38, X). It is also mutable so you can change the contents of an existing object. This helps reduce the cost of new() and garbage collection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5923) SQL std auth - parser changes
[
https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860863#comment-13860863
]
Hive QA commented on HIVE-5923:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12621039/HIVE-5923.3.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4821 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_role_grant1
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/782/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/782/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12621039
SQL std auth - parser changes
-
Key: HIVE-5923
URL: https://issues.apache.org/jira/browse/HIVE-5923
Project: Hive
Issue Type: Sub-task
Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-5923.1.patch, HIVE-5923.2.patch, HIVE-5923.3.patch
Original Estimate: 96h
Time Spent: 72h
Remaining Estimate: 12h
There are new access control statements proposed in the functional spec in
HIVE-5837 . It also proposes some small changes to the existing query syntax
(mostly extensions and some optional keywords).
The syntax supported should depend on the current authorization mode.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860872#comment-13860872
]
Thejas M Nair commented on HIVE-5795:
-
+1
Hive should be able to skip header and footer rows when reading data file for
a table
-
Key: HIVE-5795
URL: https://issues.apache.org/jira/browse/HIVE-5795
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch, HIVE-5795.3.patch,
HIVE-5795.4.patch, HIVE-5795.5.patch
Hive should be able to skip header and footer lines when reading data file
from table. In this way, user don't need to processing data which generated
by other application with a header or footer and directly use the file for
table operations.
To implement this, the idea is adding new properties in table descriptions to
define the number of lines in header and footer and skip them when reading
the record from record reader. An DDL example for creating a table with
header and footer should be like this:
{code}
Create external table testtable (name string, message string) row format
delimited fields terminated by '\t' lines terminated by '\n' location
'/testtable' tblproperties (skip.header.line.count=1,
skip.footer.line.count=2);
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Switch Precommit builds to Java 7
Sounds good to me. On Thu, Jan 2, 2014 at 10:27 AM, Brock Noland br...@cloudera.com wrote: Java 6 has been EOL for almost a year now and 7 is EOL in a short year and a quarter[1]. Therefore, I think it's time to switch the Precommit builds to Java 7. Brock 1. http://www.oracle.com/technetwork/java/eol-135779.html -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Updated] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5923: Attachment: HIVE-5923.4.patch HIVE-5923.4.patch - updating test output for new changes in trunk (HIVE-5414) SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch, HIVE-5923.2.patch, HIVE-5923.3.patch, HIVE-5923.4.patch Original Estimate: 96h Time Spent: 72h Remaining Estimate: 12h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5923: Status: Patch Available (was: Open) SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch, HIVE-5923.2.patch, HIVE-5923.3.patch, HIVE-5923.4.patch Original Estimate: 96h Time Spent: 72h Remaining Estimate: 12h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5923: Status: Open (was: Patch Available) SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch, HIVE-5923.2.patch, HIVE-5923.3.patch, HIVE-5923.4.patch Original Estimate: 96h Time Spent: 72h Remaining Estimate: 12h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860891#comment-13860891
]
Eric Hanson commented on HIVE-5795:
---
+1
Hive should be able to skip header and footer rows when reading data file for
a table
-
Key: HIVE-5795
URL: https://issues.apache.org/jira/browse/HIVE-5795
Project: Hive
Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch, HIVE-5795.3.patch,
HIVE-5795.4.patch, HIVE-5795.5.patch
Hive should be able to skip header and footer lines when reading data file
from table. In this way, user don't need to processing data which generated
by other application with a header or footer and directly use the file for
table operations.
To implement this, the idea is adding new properties in table descriptions to
define the number of lines in header and footer and skip them when reading
the record from record reader. An DDL example for creating a table with
header and footer should be like this:
{code}
Create external table testtable (name string, message string) row format
delimited fields terminated by '\t' lines terminated by '\n' location
'/testtable' tblproperties (skip.header.line.count=1,
skip.footer.line.count=2);
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive
[ https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-6017: -- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thank you Microsoft and Hideaki! Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive --- Key: HIVE-6017 URL: https://issues.apache.org/jira/browse/HIVE-6017 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: HIVE-6017.01.patch, HIVE-6017.02.patch, HIVE-6017.03.patch, HIVE-6017.04.patch Contribute the Decimal128 high-performance decimal package developed by Microsoft to Hive. This was originally written for Microsoft PolyBase by Hideaki Kimura. This code is about 8X more efficient than Java BigDecimal for typical operations. It uses a finite (128 bit) precision and can handle up to decimal(38, X). It is also mutable so you can change the contents of an existing object. This helps reduce the cost of new() and garbage collection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-3746) Fix HS2 ResultSet Serialization Performance Regression
[ https://issues.apache.org/jira/browse/HIVE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860959#comment-13860959 ] Vaibhav Gumashta commented on HIVE-3746: [~cwsteinbach] I can take a stab at what you pointed out (CLIService referencing Thrift). Otherwise, the patch looks good! Fix HS2 ResultSet Serialization Performance Regression -- Key: HIVE-3746 URL: https://issues.apache.org/jira/browse/HIVE-3746 Project: Hive Issue Type: Sub-task Components: HiveServer2, Server Infrastructure Reporter: Carl Steinbach Assignee: Navis Labels: HiveServer2, jdbc, thrift Attachments: HIVE-3746.1.patch.txt, HIVE-3746.2.patch.txt, HIVE-3746.3.patch.txt, HIVE-3746.4.patch.txt, HIVE-3746.5.patch.txt, HIVE-3746.6.patch.txt, HIVE-3746.7.patch.txt -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-3611) Hive JDBC doesn't support BINARY column
[
https://issues.apache.org/jira/browse/HIVE-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860960#comment-13860960
]
Hive QA commented on HIVE-3611:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12621147/HIVE-3611.2.patch
{color:green}SUCCESS:{color} +1 4820 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/783/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/783/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12621147
Hive JDBC doesn't support BINARY column
---
Key: HIVE-3611
URL: https://issues.apache.org/jira/browse/HIVE-3611
Project: Hive
Issue Type: Improvement
Components: JDBC
Affects Versions: 0.9.0
Reporter: Viji
Assignee: Xuefu Zhang
Labels: patch
Attachments: HIVE-3611.1.patch.txt, HIVE-3611.2.patch
Similar to HIVE-2957, the JDBC HiveDriver should also support BINARY type
columns.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Resolved] (HIVE-5532) In Windows, MapredLocalTask need to remove quotes on environment variable before launch ExecDriver in another JVM
[
https://issues.apache.org/jira/browse/HIVE-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuaishuai Nie resolved HIVE-5532.
--
Resolution: Won't Fix
In Windows, MapredLocalTask need to remove quotes on environment variable
before launch ExecDriver in another JVM
-
Key: HIVE-5532
URL: https://issues.apache.org/jira/browse/HIVE-5532
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-5532.1.patch
In Windows environment, environment variable is quoted to preserve special
characters like space. However since MapredLocalTask will call hadoop.cmd to
launch ExecDriver and hadoop.cmd will reconstruct environment variable like
{code}
set HADOOP_OPTS=%HADOOP_OPTS% %HADOOP_CLIENT_OPTS%
{code}
the quotes for HADOOP_CLIENT_OPTS will be part of HADOOP_OPTS and nullify the
variables in HADOOP_CLIENT_OPTS after appending to HADOOP_OPTS.
Example HADOOP_OPTS:
{code}
-Dhadoop.log.dir=C:\apps\dist\hadoop-1.2.0.1.3.1.0-06\logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=C:\apps\dist\hadoop-1.2.0.1.3.1.0-06
-Dhadoop.root.logger=INFO,TLA
-Djava.library.path=;C:\apps\dist\hadoop-1.2.0.1.3.1.0-06\lib\native\Windows_NT-amd64-64;C:\apps\dist\hadoop-1.2.0.1.3.1.0-06\lib\native
-Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.tasklog.taskid=attempt_201310092133_0006_m_00_0
-Dhadoop.tasklog.iscleanup=false -Dhadoop.tasklog.totalLogFileSize=0
{code}
One failure scenario is when map side join is launched by Oozie, the job
fails because cannot find hadoop.tasklog.taskid and throw exception when
initializing log4j
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5532) In Windows, MapredLocalTask need to remove quotes on environment variable before launch ExecDriver in another JVM
[
https://issues.apache.org/jira/browse/HIVE-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860995#comment-13860995
]
Shuaishuai Nie commented on HIVE-5532:
--
Thanks @Thejas, closing this as won't fix in hive based on the comment
In Windows, MapredLocalTask need to remove quotes on environment variable
before launch ExecDriver in another JVM
-
Key: HIVE-5532
URL: https://issues.apache.org/jira/browse/HIVE-5532
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-5532.1.patch
In Windows environment, environment variable is quoted to preserve special
characters like space. However since MapredLocalTask will call hadoop.cmd to
launch ExecDriver and hadoop.cmd will reconstruct environment variable like
{code}
set HADOOP_OPTS=%HADOOP_OPTS% %HADOOP_CLIENT_OPTS%
{code}
the quotes for HADOOP_CLIENT_OPTS will be part of HADOOP_OPTS and nullify the
variables in HADOOP_CLIENT_OPTS after appending to HADOOP_OPTS.
Example HADOOP_OPTS:
{code}
-Dhadoop.log.dir=C:\apps\dist\hadoop-1.2.0.1.3.1.0-06\logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=C:\apps\dist\hadoop-1.2.0.1.3.1.0-06
-Dhadoop.root.logger=INFO,TLA
-Djava.library.path=;C:\apps\dist\hadoop-1.2.0.1.3.1.0-06\lib\native\Windows_NT-amd64-64;C:\apps\dist\hadoop-1.2.0.1.3.1.0-06\lib\native
-Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.tasklog.taskid=attempt_201310092133_0006_m_00_0
-Dhadoop.tasklog.iscleanup=false -Dhadoop.tasklog.totalLogFileSize=0
{code}
One failure scenario is when map side join is launched by Oozie, the job
fails because cannot find hadoop.tasklog.taskid and throw exception when
initializing log4j
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Hive-trunk-hadoop2 - Build # 643 - Still Failing
Changes for Build #640 Changes for Build #641 [navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis reviewed by Thejas M Nair) [navis] HIVE-4257 : java.sql.SQLNonTransientConnectionException on JDBCStatsAggregator (Teddy Choi via Navis, reviewed by Ashutosh) Changes for Build #642 Changes for Build #643 [ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive (Hideaki Kumura via Eric Hanson) No tests ran. The Apache Jenkins build system has built Hive-trunk-hadoop2 (build #643) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-hadoop2/643/ to view the results.
[jira] [Commented] (HIVE-6051) Create DecimalColumnVector and a representative VectorExpression for decimal
[ https://issues.apache.org/jira/browse/HIVE-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860999#comment-13860999 ] Eric Hanson commented on HIVE-6051: --- Review at https://reviews.apache.org/r/16584/ Create DecimalColumnVector and a representative VectorExpression for decimal Key: HIVE-6051 URL: https://issues.apache.org/jira/browse/HIVE-6051 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: HIVE-6051.01.patch Create a DecimalColumnVector to use as a basis for vectorized decimal operations. Include a representative VectorExpression on decimal (e.g. column-column addition) to demonstrate it's use. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6051) Create DecimalColumnVector and a representative VectorExpression for decimal
[ https://issues.apache.org/jira/browse/HIVE-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-6051: -- Fix Version/s: 0.13.0 Status: Patch Available (was: In Progress) Create DecimalColumnVector and a representative VectorExpression for decimal Key: HIVE-6051 URL: https://issues.apache.org/jira/browse/HIVE-6051 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: HIVE-6051.01.patch Create a DecimalColumnVector to use as a basis for vectorized decimal operations. Include a representative VectorExpression on decimal (e.g. column-column addition) to demonstrate it's use. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-h0.21 - Build # 2542 - Still Failing
Changes for Build #2539 Changes for Build #2540 [navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis reviewed by Thejas M Nair) Changes for Build #2541 Changes for Build #2542 [ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive (Hideaki Kumura via Eric Hanson) No tests ran. The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2542) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2542/ to view the results.
Review Request 16593: HIVE-3611: Hive JDBC doesn't support BINARY column
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16593/ --- Review request for hive. Bugs: HIVE-3611 https://issues.apache.org/jira/browse/HIVE-3611 Repository: hive-git Description --- Added BINARY data type support in JDBC2. Diffs - data/files/datatypes.txt 1e48c8d itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 4090e72 jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java b02f374 jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 3116100 jdbc/src/java/org/apache/hive/jdbc/HiveResultSetMetaData.java e89571f jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java 42ec32a jdbc/src/java/org/apache/hive/jdbc/Utils.java 913dc46 Diff: https://reviews.apache.org/r/16593/diff/ Testing --- Changed existing test cases to include binary data type in TestJdbcDriver2. Thanks, Xuefu Zhang
[jira] [Commented] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861020#comment-13861020 ] Navis commented on HIVE-4095: - [~leftylev] I think the document is right. It might induce backward compatibility issue but this should be fixed. I'm on it. Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Fix For: 0.12.0 Attachments: HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, hive.4095.1.patch, hive.4095.refresh.patch, hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6129) alter exchange is implemented in inverted manner
Navis created HIVE-6129:
---
Summary: alter exchange is implemented in inverted manner
Key: HIVE-6129
URL: https://issues.apache.org/jira/browse/HIVE-6129
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Critical
alter exchange should be implemented accord to document in
https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition. i.e
{code}
alter table T1 exchange partition (ds='1') with table T2
{code}
should be (after creating T1@ds=1)
{quote}
moves the data from T2 to T1@ds=1
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6129) alter exchange is implemented in inverted manner
[
https://issues.apache.org/jira/browse/HIVE-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-6129:
Description:
see
https://issues.apache.org/jira/browse/HIVE-4095?focusedCommentId=13819885page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13819885
alter exchange should be implemented accord to document in
https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition. i.e
{code}
alter table T1 exchange partition (ds='1') with table T2
{code}
should be (after creating T1@ds=1)
{quote}
moves the data from T2 to T1@ds=1
{quote}
was:
alter exchange should be implemented accord to document in
https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition. i.e
{code}
alter table T1 exchange partition (ds='1') with table T2
{code}
should be (after creating T1@ds=1)
{quote}
moves the data from T2 to T1@ds=1
{quote}
alter exchange is implemented in inverted manner
Key: HIVE-6129
URL: https://issues.apache.org/jira/browse/HIVE-6129
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Critical
see
https://issues.apache.org/jira/browse/HIVE-4095?focusedCommentId=13819885page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13819885
alter exchange should be implemented accord to document in
https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition. i.e
{code}
alter table T1 exchange partition (ds='1') with table T2
{code}
should be (after creating T1@ds=1)
{quote}
moves the data from T2 to T1@ds=1
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Review Request 16229: HIVE-6010 create a test that would ensure vectorization produces same results as non-vectorized execution
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16229/#review31098 --- Ship it! Ship It! - Jitendra Pandey On Dec. 17, 2013, 10:40 p.m., Sergey Shelukhin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16229/ --- (Updated Dec. 17, 2013, 10:40 p.m.) Review request for hive and Jitendra Pandey. Bugs: HIVE-6010 https://issues.apache.org/jira/browse/HIVE-6010 Repository: hive-git Description --- See jira. Diffs - ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 79840c9 itests/qtest/pom.xml 971c5d3 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 275e3d7 ql/src/test/queries/clientcompare/vectorized_math_funcs.q PRE-CREATION ql/src/test/queries/clientcompare/vectorized_math_funcs_00.qv PRE-CREATION ql/src/test/queries/clientcompare/vectorized_math_funcs_01.qv PRE-CREATION ql/src/test/templates/TestCompareCliDriver.vm PRE-CREATION Diff: https://reviews.apache.org/r/16229/diff/ Testing --- Thanks, Sergey Shelukhin
[jira] [Updated] (HIVE-6010) create a test that would ensure vectorization produces same results as non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6010: --- Resolution: Fixed Fix Version/s: 0.13.0 Release Note: Committed to trunk. Thanks Sergey! Status: Resolved (was: Patch Available) create a test that would ensure vectorization produces same results as non-vectorized execution --- Key: HIVE-6010 URL: https://issues.apache.org/jira/browse/HIVE-6010 Project: Hive Issue Type: Test Components: Tests, Vectorization Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6010.01.patch, HIVE-6010.02.patch, HIVE-6010.03.patch, HIVE-6010.04.patch, HIVE-6010.patch So as to ensure that vectorization is not forgotten when changes are made to things. Obviously it would not be viable to have a bulletproof test, but at least a subset of operations can be verified. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-3746) Fix HS2 ResultSet Serialization Performance Regression
[ https://issues.apache.org/jira/browse/HIVE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861034#comment-13861034 ] Carl Steinbach commented on HIVE-3746: -- [~vaibhavgumashta] Cool! [~navis] Thanks again for putting this patch together! I'm in the process of committing it now. Fix HS2 ResultSet Serialization Performance Regression -- Key: HIVE-3746 URL: https://issues.apache.org/jira/browse/HIVE-3746 Project: Hive Issue Type: Sub-task Components: HiveServer2, Server Infrastructure Reporter: Carl Steinbach Assignee: Navis Labels: HiveServer2, jdbc, thrift Attachments: HIVE-3746.1.patch.txt, HIVE-3746.2.patch.txt, HIVE-3746.3.patch.txt, HIVE-3746.4.patch.txt, HIVE-3746.5.patch.txt, HIVE-3746.6.patch.txt, HIVE-3746.7.patch.txt -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6010) create a test that would ensure vectorization produces same results as non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6010: --- Release Note: (was: Committed to trunk. Thanks Sergey!) Removing release note Committed to trunk. Thanks Sergey! :) create a test that would ensure vectorization produces same results as non-vectorized execution --- Key: HIVE-6010 URL: https://issues.apache.org/jira/browse/HIVE-6010 Project: Hive Issue Type: Test Components: Tests, Vectorization Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6010.01.patch, HIVE-6010.02.patch, HIVE-6010.03.patch, HIVE-6010.04.patch, HIVE-6010.patch So as to ensure that vectorization is not forgotten when changes are made to things. Obviously it would not be viable to have a bulletproof test, but at least a subset of operations can be verified. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-3611) Hive JDBC doesn't support BINARY column
[ https://issues.apache.org/jira/browse/HIVE-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861045#comment-13861045 ] Wee Hiong commented on HIVE-3611: - Hi Xuefu, So glad to hear the patch is available now. I need some instruction on how to install this patch into my Hadoop cluster. Kindly advise... Hive JDBC doesn't support BINARY column --- Key: HIVE-3611 URL: https://issues.apache.org/jira/browse/HIVE-3611 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.9.0 Reporter: Viji Assignee: Xuefu Zhang Labels: patch Attachments: HIVE-3611.1.patch.txt, HIVE-3611.2.patch Similar to HIVE-2957, the JDBC HiveDriver should also support BINARY type columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6130) Webhcat resource /templeton/v1/status doesn't track Hive job progress
Shuaishuai Nie created HIVE-6130: Summary: Webhcat resource /templeton/v1/status doesn't track Hive job progress Key: HIVE-6130 URL: https://issues.apache.org/jira/browse/HIVE-6130 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Problem: Unable to track Hive job progress through webhcat. Cause: TempletonUtils has code to parse PIG and JAR child jobid and percentage progress but none for Hive jobs. extractPercentComplete() and extractChildJobId() are supposed to do this parsing. Effect: /templeton/v1/queue/$job_id?user.name=$user returns no job progress info. The jobid param in the json result that's supposed to contain the child hive jobid contains the templetoncontrollerjob id instead leaving the parent jobid null. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6131) New columns after table alter result in null values despite data
James Vaughan created HIVE-6131: --- Summary: New columns after table alter result in null values despite data Key: HIVE-6131 URL: https://issues.apache.org/jira/browse/HIVE-6131 Project: Hive Issue Type: Bug Reporter: James Vaughan Priority: Minor Hi folks, I found and verified a bug on our CDH 4.0.3 install of Hive when adding columns to tables with Partitions using 'REPLACE COLUMNS'. I dug through the Jira a little bit and didn't see anything for it so hopefully this isn't just noise on the radar. Basically, when you alter a table with partitions and then reupload data to that partition, it doesn't seem to recognize the extra data that actually exists in HDFS- as in, returns NULL values on the new column despite having the data and recognizing the new column in the metadata. Here's some steps to reproduce using a basic table: 1. Run this hive command: CREATE TABLE jvaughan_test (col1 string) partitioned by (day string); 2. Create a simple file on the system with a couple of entries, something like hi and hi2 separated by newlines. 3. Run this hive command, pointing it at the file: LOAD DATA LOCAL INPATH 'FILEDIR' OVERWRITE INTO TABLE jvaughan_test PARTITION (day = '2014-01-02'); 4. Confirm the data with: SELECT * FROM jvaughan_test WHERE day = '2014-01-02'; 5. Alter the column definitions: ALTER TABLE jvaughan_test REPLACE COLUMNS (col1 string, col2 string); 6. Edit your file and add a second column using the default separator (ctrl+v, then ctrl+a in Vim) and add two more entries, such as hi3 on the first row and hi4 on the second 7. Run step 3 again 8. Check the data again like in step 4 For me, this is the results that get returned: hive select * from jvaughan_test where day = '2014-01-01'; OK hi NULL2014-01-02 hi2 NULL2014-01-02 This is despite the fact that there is data in the file stored by the partition in HDFS. Let me know if you need any other information. The only workaround for me currently is to drop partitions for any I'm replacing data in and THEN reupload the new data file. Thanks, -James -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6130) Webhcat resource /templeton/v1/status doesn't track Hive job progress
[ https://issues.apache.org/jira/browse/HIVE-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861060#comment-13861060 ] Shuaishuai Nie commented on HIVE-6130: -- Create this JIRA to keep tracking of HCATALOG-647. [~gautamk] can you please re-base the patch in HCATALOG-647 to hive trunk? This is an important fix and we want to get this fix committed. If it is inconvenience for you, I can help you with it. Thanks Webhcat resource /templeton/v1/status doesn't track Hive job progress - Key: HIVE-6130 URL: https://issues.apache.org/jira/browse/HIVE-6130 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Problem: Unable to track Hive job progress through webhcat. Cause: TempletonUtils has code to parse PIG and JAR child jobid and percentage progress but none for Hive jobs. extractPercentComplete() and extractChildJobId() are supposed to do this parsing. Effect: /templeton/v1/queue/$job_id?user.name=$user returns no job progress info. The jobid param in the json result that's supposed to contain the child hive jobid contains the templetoncontrollerjob id instead leaving the parent jobid null. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5923) SQL std auth - parser changes
[
https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861068#comment-13861068
]
Hive QA commented on HIVE-5923:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12621175/HIVE-5923.4.patch
{color:green}SUCCESS:{color} +1 4872 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/785/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/785/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12621175
SQL std auth - parser changes
-
Key: HIVE-5923
URL: https://issues.apache.org/jira/browse/HIVE-5923
Project: Hive
Issue Type: Sub-task
Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-5923.1.patch, HIVE-5923.2.patch, HIVE-5923.3.patch,
HIVE-5923.4.patch
Original Estimate: 96h
Time Spent: 72h
Remaining Estimate: 12h
There are new access control statements proposed in the functional spec in
HIVE-5837 . It also proposes some small changes to the existing query syntax
(mostly extensions and some optional keywords).
The syntax supported should depend on the current authorization mode.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6130) Webhcat resource /templeton/v1/status doesn't track Hive job progress
[ https://issues.apache.org/jira/browse/HIVE-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861071#comment-13861071 ] Eugene Koifman commented on HIVE-6130: -- TempletonUtils already has logic for Hive percentComplete. Also, inspite of the title for this bug, I assume you are not planning to change what ./templeton/v1/status method does, correct? Webhcat resource /templeton/v1/status doesn't track Hive job progress - Key: HIVE-6130 URL: https://issues.apache.org/jira/browse/HIVE-6130 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Problem: Unable to track Hive job progress through webhcat. Cause: TempletonUtils has code to parse PIG and JAR child jobid and percentage progress but none for Hive jobs. extractPercentComplete() and extractChildJobId() are supposed to do this parsing. Effect: /templeton/v1/queue/$job_id?user.name=$user returns no job progress info. The jobid param in the json result that's supposed to contain the child hive jobid contains the templetoncontrollerjob id instead leaving the parent jobid null. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6124) Support basic Decimal arithmetic in vector mode (+, -, *)
[ https://issues.apache.org/jira/browse/HIVE-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-6124: -- Attachment: HIVE-6124.01.patch Partially finished patch, parked here for safe-keeping. Supports decimal column OP decimal scalar, and includes unit tests. Currently includes DecimalColumnVector but I'll remove that before commit. Still need to do scalar-op-col and col-op-col templates for decimal. Support basic Decimal arithmetic in vector mode (+, -, *) - Key: HIVE-6124 URL: https://issues.apache.org/jira/browse/HIVE-6124 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-6124.01.patch Create support for basic decimal arithmetic (+, -, * but not /, %) based on templates for column-scalar, scalar-column, and column-column operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-3746) Fix HS2 ResultSet Serialization Performance Regression
[ https://issues.apache.org/jira/browse/HIVE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3746: - Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks Navis! Fix HS2 ResultSet Serialization Performance Regression -- Key: HIVE-3746 URL: https://issues.apache.org/jira/browse/HIVE-3746 Project: Hive Issue Type: Sub-task Components: HiveServer2, Server Infrastructure Reporter: Carl Steinbach Assignee: Navis Labels: HiveServer2, jdbc, thrift Fix For: 0.13.0 Attachments: HIVE-3746.1.patch.txt, HIVE-3746.2.patch.txt, HIVE-3746.3.patch.txt, HIVE-3746.4.patch.txt, HIVE-3746.5.patch.txt, HIVE-3746.6.patch.txt, HIVE-3746.7.patch.txt -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6017) Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive
[ https://issues.apache.org/jira/browse/HIVE-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861085#comment-13861085 ] Lefty Leverenz commented on HIVE-6017: -- What documentation does this need -- just add DECIMAL128 to the list of types with a brief explanation? * [Numeric Types|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-NumericTypes] * [Decimals|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-Decimals] * [Floating Point Types|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27838462#LanguageManualTypes-FloatingPointTypes] Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive --- Key: HIVE-6017 URL: https://issues.apache.org/jira/browse/HIVE-6017 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: HIVE-6017.01.patch, HIVE-6017.02.patch, HIVE-6017.03.patch, HIVE-6017.04.patch Contribute the Decimal128 high-performance decimal package developed by Microsoft to Hive. This was originally written for Microsoft PolyBase by Hideaki Kimura. This code is about 8X more efficient than Java BigDecimal for typical operations. It uses a finite (128 bit) precision and can handle up to decimal(38, X). It is also mutable so you can change the contents of an existing object. This helps reduce the cost of new() and garbage collection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5446) Hive can CREATE an external table but not SELECT from it when file path have spaces
[ https://issues.apache.org/jira/browse/HIVE-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-5446: - Attachment: HIVE-5446.2.patch Re-based the patch on trunk Hive can CREATE an external table but not SELECT from it when file path have spaces --- Key: HIVE-5446 URL: https://issues.apache.org/jira/browse/HIVE-5446 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-5446.1.patch, HIVE-5446.2.patch Create external table table1 (age int, gender string, totBil float, dirBill float, alkphos int, sgpt int, sgot int, totProt float, aLB float, aG float, sel int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION 'hdfs://namenodehost:9000/hive newtable'; select * from table1; return nothing even there is file in the target folder -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6132) Support hbase filters for Hive HBase Integration
Swarnim Kulkarni created HIVE-6132: -- Summary: Support hbase filters for Hive HBase Integration Key: HIVE-6132 URL: https://issues.apache.org/jira/browse/HIVE-6132 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.12.0 Reporter: Swarnim Kulkarni Along with predicate pushdown to optimize HBase table scans, we should support HBase filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5911) Recent change to schema upgrade scripts breaks file naming conventions
[ https://issues.apache.org/jira/browse/HIVE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5911: - Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks Sergey! Recent change to schema upgrade scripts breaks file naming conventions -- Key: HIVE-5911 URL: https://issues.apache.org/jira/browse/HIVE-5911 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-5911.01.patch, HIVE-5911.patch The changes made in HIVE-5700 break the convention for naming schema upgrade scripts. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-h0.21 - Build # 2543 - Still Failing
Changes for Build #2539 Changes for Build #2540 [navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis reviewed by Thejas M Nair) Changes for Build #2541 Changes for Build #2542 [ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive (Hideaki Kumura via Eric Hanson) Changes for Build #2543 [cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II (Navis via cws) [cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis via cws) [jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure vectorization produces same results as non-vectorized execution (Sergey Shelukhin via Jitendra Pandey) No tests ran. The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2543) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2543/ to view the results.
Hive-trunk-hadoop2 - Build # 644 - Still Failing
Changes for Build #640 Changes for Build #641 [navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis reviewed by Thejas M Nair) [navis] HIVE-4257 : java.sql.SQLNonTransientConnectionException on JDBCStatsAggregator (Teddy Choi via Navis, reviewed by Ashutosh) Changes for Build #642 Changes for Build #643 [ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive (Hideaki Kumura via Eric Hanson) Changes for Build #644 [cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming conventions (Sergey Shelukhin via cws) [cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II (Navis via cws) [cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis via cws) [jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure vectorization produces same results as non-vectorized execution (Sergey Shelukhin via Jitendra Pandey) No tests ran. The Apache Jenkins build system has built Hive-trunk-hadoop2 (build #644) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-hadoop2/644/ to view the results.
[jira] [Updated] (HIVE-5224) When creating table with AVRO serde, the avro.schema.url should be about to load serde schema from file system beside HDFS
[ https://issues.apache.org/jira/browse/HIVE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-5224: - Attachment: HIVE-5224.4.patch Upload the same patch to verify through Hive QA When creating table with AVRO serde, the avro.schema.url should be about to load serde schema from file system beside HDFS Key: HIVE-5224 URL: https://issues.apache.org/jira/browse/HIVE-5224 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-5224.1.patch, HIVE-5224.2.patch, HIVE-5224.4.patch, Hive-5224.3.patch Now when loading schema for table with AVRO serde, the file system is hard coded to hdfs in AvroSerdeUtils.java. This should enable loading schema from file system beside hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Hive-trunk-h0.21 - Build # 2544 - Still Failing
Changes for Build #2539 Changes for Build #2540 [navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis reviewed by Thejas M Nair) Changes for Build #2541 Changes for Build #2542 [ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package from Microsoft to Hive (Hideaki Kumura via Eric Hanson) Changes for Build #2543 [cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II (Navis via cws) [cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis via cws) [jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure vectorization produces same results as non-vectorized execution (Sergey Shelukhin via Jitendra Pandey) Changes for Build #2544 [cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming conventions (Sergey Shelukhin via cws) No tests ran. The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2544) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2544/ to view the results.
[jira] [Updated] (HIVE-5032) Enable hive creating external table at the root directory of DFS
[ https://issues.apache.org/jira/browse/HIVE-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-5032: - Attachment: HIVE-5032.3.patch Re-based the patch on latest trunk. Enable hive creating external table at the root directory of DFS Key: HIVE-5032 URL: https://issues.apache.org/jira/browse/HIVE-5032 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-5032.1.patch, HIVE-5032.2.patch, HIVE-5032.3.patch Creating external table using HIVE with location point to the root directory of DFS will fail because the function HiveFileFormatUtils#doGetPartitionDescFromPath treat authority of the path the same as folder and cannot find a match in the pathToPartitionInfo table when doing prefix match. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-4773) webhcat intermittently fail to commit output to file system
[ https://issues.apache.org/jira/browse/HIVE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-4773: - Attachment: HIVE-4773.4.patch Re-based the patch on latest trunk. webhcat intermittently fail to commit output to file system --- Key: HIVE-4773 URL: https://issues.apache.org/jira/browse/HIVE-4773 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-4773.1.patch, HIVE-4773.2.patch, HIVE-4773.3.patch, HIVE-4773.4.patch With ASV as a default FS, we saw instances where output is not fully flushed to storage before the Templeton controller process exits. This results in stdout and stderr being empty even though the job completed successfully. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-4760) webhcat occasional raises bad request (HTTP error 400) for queue/:jobid requests
[ https://issues.apache.org/jira/browse/HIVE-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-4760: - Resolution: Not A Problem Status: Resolved (was: Patch Available) Closing this JIRA as Not A Problem since the function JobState.getChildId() is deprecated in the latest version of trunk webhcat occasional raises bad request (HTTP error 400) for queue/:jobid requests Key: HIVE-4760 URL: https://issues.apache.org/jira/browse/HIVE-4760 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-4760.1.patch This issue randomly occurs. The repro we have found is to issue a job and then write a loop that calls queue/:jobid over and over again until the job completes. At some point depending on the timing, a bad request (jobid is not valid) - is raised. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Review Request 16565: HIVE-5923 - SQL std auth - parser changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16565/ --- (Updated Jan. 3, 2014, 2:46 a.m.) Review request for hive and Brock Noland. Changes --- HIVE-5923.4.patch - updating test output for new changes in trunk (HIVE-5414) Repository: hive-git Description --- Grant privilege and revoke privilege statements need to be changed to remove the requirement (but not the option) for the noise word TABLE. In the SQL specification table is the assumed default for grant and revoke statements. Today Hive’s syntax is GRANT action ON TABLE table TO grantee. It should be GRANT action ON [TABLE] table TO grantee. Grant role and revoke role statements has been changed to remove the need for keyword ROLE. Support for WITH ADMIN OPTION needs to be added to grant role and revoke role statement syntax. HIVE-5923.3.patch Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 8d0b7a8 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java a1b19a6 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 9354637 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g b34f53b ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 39d53d3 ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java f0acb16 ql/src/test/queries/clientpositive/authorization_2.q 4fc79b9 ql/src/test/queries/clientpositive/authorization_role_grant1.q PRE-CREATION ql/src/test/results/clientpositive/authorization_2.q.out 460240b ql/src/test/results/clientpositive/authorization_role_grant1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/16565/diff/ Testing --- Thanks, Thejas Nair
[jira] [Commented] (HIVE-5923) SQL std auth - parser changes
[ https://issues.apache.org/jira/browse/HIVE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861137#comment-13861137 ] Thejas M Nair commented on HIVE-5923: - [~brocknoland] Can you please take a look at this minor update to the patch as well (test case output fix)? SQL std auth - parser changes - Key: HIVE-5923 URL: https://issues.apache.org/jira/browse/HIVE-5923 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-5923.1.patch, HIVE-5923.2.patch, HIVE-5923.3.patch, HIVE-5923.4.patch Original Estimate: 96h Time Spent: 72h Remaining Estimate: 12h There are new access control statements proposed in the functional spec in HIVE-5837 . It also proposes some small changes to the existing query syntax (mostly extensions and some optional keywords). The syntax supported should depend on the current authorization mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Review Request 16600: alter exchange is implemented in inverted manner
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16600/
---
Review request for hive.
Bugs: HIVE-6129
https://issues.apache.org/jira/browse/HIVE-6129
Repository: hive-git
Description
---
see
https://issues.apache.org/jira/browse/HIVE-4095?focusedCommentId=13819885page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13819885
alter exchange should be implemented accord to document in
https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition. i.e
{code}
alter table T1 exchange partition (ds='1') with table T2
{code}
should be (after creating T1@ds=1)
{quote}
moves the data from T2 to T1@ds=1
{quote}
Diffs
-
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java a1b19a6
ql/src/test/queries/clientnegative/exchange_partition_neg_incomplete_partition.q
e4f0dac
ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists.q
4d1e0a6
ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists2.q
23777db
ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists3.q
350bf24
ql/src/test/queries/clientnegative/exchange_partition_neg_partition_missing.q
81944b3
ql/src/test/queries/clientpositive/exchange_partition.q 6e8bf8e
ql/src/test/queries/clientpositive/exchange_partition2.q 27b335a
ql/src/test/queries/clientpositive/exchange_partition3.q 7b9060d
ql/src/test/results/clientnegative/exchange_partition_neg_incomplete_partition.q.out
8499427
ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists.q.out
b8c3c139
ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists2.q.out
93c3406
ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists3.q.out
d0959ef
ql/src/test/results/clientpositive/exchange_partition.q.out bbc6d5b
ql/src/test/results/clientpositive/exchange_partition2.q.out 5136106
ql/src/test/results/clientpositive/exchange_partition3.q.out 72ac6bd
Diff: https://reviews.apache.org/r/16600/diff/
Testing
---
Thanks,
Navis Ryu
[jira] [Updated] (HIVE-6129) alter exchange is implemented in inverted manner
[
https://issues.apache.org/jira/browse/HIVE-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-6129:
Status: Patch Available (was: Open)
alter exchange is implemented in inverted manner
Key: HIVE-6129
URL: https://issues.apache.org/jira/browse/HIVE-6129
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: HIVE-6129.1.patch.txt
see
https://issues.apache.org/jira/browse/HIVE-4095?focusedCommentId=13819885page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13819885
alter exchange should be implemented accord to document in
https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition. i.e
{code}
alter table T1 exchange partition (ds='1') with table T2
{code}
should be (after creating T1@ds=1)
{quote}
moves the data from T2 to T1@ds=1
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6129) alter exchange is implemented in inverted manner
[
https://issues.apache.org/jira/browse/HIVE-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-6129:
Attachment: HIVE-6129.1.patch.txt
alter exchange is implemented in inverted manner
Key: HIVE-6129
URL: https://issues.apache.org/jira/browse/HIVE-6129
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Critical
Attachments: HIVE-6129.1.patch.txt
see
https://issues.apache.org/jira/browse/HIVE-4095?focusedCommentId=13819885page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13819885
alter exchange should be implemented accord to document in
https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition. i.e
{code}
alter table T1 exchange partition (ds='1') with table T2
{code}
should be (after creating T1@ds=1)
{quote}
moves the data from T2 to T1@ds=1
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6105) LongWritable.compareTo needs shimming
[
https://issues.apache.org/jira/browse/HIVE-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861198#comment-13861198
]
Hive QA commented on HIVE-6105:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12621031/HIVE-6105.1.patch.txt
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4874 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/786/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/786/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12621031
LongWritable.compareTo needs shimming
-
Key: HIVE-6105
URL: https://issues.apache.org/jira/browse/HIVE-6105
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Gunther Hagleitner
Assignee: Navis
Attachments: HIVE-6105.1.patch.txt
Compiled hive against hadoop-2. Running tests on hadoop 1 gives:
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.io.LongWritable.compareTo(Lorg/apache/hadoop/io/LongWritable;)I
at
org.apache.hadoop.hive.ql.udf.UDAFPercentile$MyComparator.compare(UDAFPercentile.java:62)
at
org.apache.hadoop.hive.ql.udf.UDAFPercentile$MyComparator.compare(UDAFPercentile.java:58)
at java.util.Arrays.mergeSort(Arrays.java:1270)
at java.util.Arrays.sort(Arrays.java:1210)
at java.util.Collections.sort(Collections.java:157)
at
org.apache.hadoop.hive.ql.udf.UDAFPercentile$PercentileLongEvaluator.terminate(UDAFPercentile.java:194)
... 20 more
I think the problem is that the compareTo function has changed from using
Object to using LongWritable as an argument. Seems this needs a shim.
However, we should also make sure that this shim doesn't hit performance too
hard.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HIVE-6133) Support partial partition exchange
Navis created HIVE-6133:
---
Summary: Support partial partition exchange
Key: HIVE-6133
URL: https://issues.apache.org/jira/browse/HIVE-6133
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Current alter exchange coerces source and destination table to have same
partition columns. But source table has sub-set of partitions and provided
partition spec supplements to be a complete partition spec, it need not to be
that.
For example,
{nofomat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING);
CREATE TABLE exchange_part_test2 (f1 string)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE
exchange_part_test2;
{nofomat}
or
{nofomat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING, hr
STRING);
CREATE TABLE exchange_part_test2 (f1 string) PARTITIONED BY (hr STRING)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE
exchange_part_test2;
{nofomat}
can be possible.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6133) Support partial partition exchange
[
https://issues.apache.org/jira/browse/HIVE-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-6133:
Description:
Current alter exchange coerces source and destination table to have same
partition columns. But source table has sub-set of partitions and provided
partition spec supplements to be a complete partition spec, it need not to be
that.
For example,
{noformat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING);
CREATE TABLE exchange_part_test2 (f1 string)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE
exchange_part_test2;
{noformat}
or
{noformat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING, hr
STRING);
CREATE TABLE exchange_part_test2 (f1 string) PARTITIONED BY (hr STRING)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE
exchange_part_test2;
{noformat}
can be possible.
was:
Current alter exchange coerces source and destination table to have same
partition columns. But source table has sub-set of partitions and provided
partition spec supplements to be a complete partition spec, it need not to be
that.
For example,
{nofomat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING);
CREATE TABLE exchange_part_test2 (f1 string)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE
exchange_part_test2;
{nofomat}
or
{nofomat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING, hr
STRING);
CREATE TABLE exchange_part_test2 (f1 string) PARTITIONED BY (hr STRING)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE
exchange_part_test2;
{nofomat}
can be possible.
Support partial partition exchange
--
Key: HIVE-6133
URL: https://issues.apache.org/jira/browse/HIVE-6133
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Current alter exchange coerces source and destination table to have same
partition columns. But source table has sub-set of partitions and provided
partition spec supplements to be a complete partition spec, it need not to be
that.
For example,
{noformat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING);
CREATE TABLE exchange_part_test2 (f1 string)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH
TABLE exchange_part_test2;
{noformat}
or
{noformat}
CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING, hr
STRING);
CREATE TABLE exchange_part_test2 (f1 string) PARTITIONED BY (hr STRING)
ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH
TABLE exchange_part_test2;
{noformat}
can be possible.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-3611) Hive JDBC doesn't support BINARY column
[ https://issues.apache.org/jira/browse/HIVE-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861231#comment-13861231 ] Xuefu Zhang commented on HIVE-3611: --- [~weehiong] The patch, while subject to review, will be committed to trunk, which means that you can get it from the next release (0.13). For your existing hive installation, however, you may choose to backport this patch to the version of hive that you have and rebuild your hive binaries. Practically, a lot of users would ask their service provides to do this for them. This certainly goes beyond the responsibility of the Hive community here. Hive JDBC doesn't support BINARY column --- Key: HIVE-3611 URL: https://issues.apache.org/jira/browse/HIVE-3611 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.9.0 Reporter: Viji Assignee: Xuefu Zhang Labels: patch Attachments: HIVE-3611.1.patch.txt, HIVE-3611.2.patch Similar to HIVE-2957, the JDBC HiveDriver should also support BINARY type columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-3611) Hive JDBC doesn't support BINARY column
[ https://issues.apache.org/jira/browse/HIVE-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861250#comment-13861250 ] Wee Hiong commented on HIVE-3611: - Hi Xuefu, OK, I understand now. I will wait for next Hive release (0.13). Thank you for the advise. Hive JDBC doesn't support BINARY column --- Key: HIVE-3611 URL: https://issues.apache.org/jira/browse/HIVE-3611 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.9.0 Reporter: Viji Assignee: Xuefu Zhang Labels: patch Attachments: HIVE-3611.1.patch.txt, HIVE-3611.2.patch Similar to HIVE-2957, the JDBC HiveDriver should also support BINARY type columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6125) Tez: Refactoring changes
[
https://issues.apache.org/jira/browse/HIVE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861299#comment-13861299
]
Hive QA commented on HIVE-6125:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12621026/HIVE-6125.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4873 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/787/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/787/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12621026
Tez: Refactoring changes
Key: HIVE-6125
URL: https://issues.apache.org/jira/browse/HIVE-6125
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-6125.1.patch
In order to facilitate merge back I've separated out all the changes that
don't require Tez. These changes introduce new interfaces, move code etc. In
preparation of the Tez specific classes. This should help show what changes
have been made that affect the MR codepath as well.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
