[jira] [Updated] (HIVE-1653) Ability to enforce correct stats
[ https://issues.apache.org/jira/browse/HIVE-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1653: - Status: Open (was: Patch Available) @Namit: Added a comment on phabricator. Thanks. Ability to enforce correct stats Key: HIVE-1653 URL: https://issues.apache.org/jira/browse/HIVE-1653 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain This is a follow-up for https://issues.apache.org/jira/browse/HIVE-1361. If one of the mappers/reducers cannot publish stats, it may lead to wrong aggregated stats. There should be a way to avoid this - at the least, a configuration variable which fails the task if stats cannot be published -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1563 - Still Failing
Changes for Build #1561 Changes for Build #1562 [kevinwilfong] HIVE-3261. alter the number of buckets for a non-empty partitioned table should not be allowed. (njain via kevinwilfong) Changes for Build #1563 [hashutosh] HIVE-3225 : NPE on a join query with authorization enabled (Francis Liu via Ashutosh Chauhan) 2 tests failed. FAILED: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10722) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) REGRESSION: org.apache.hadoop.hive.service.TestHiveServerSessions.testSessionVars Error Message: java.net.ConnectException: Connection refused Stack Trace: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:183) at org.apache.hadoop.hive.service.TestHiveServerSessions.setUp(TestHiveServerSessions.java:59) at junit.framework.TestCase.runBare(TestCase.java:132) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at org.apache.thrift.transport.TSocket.open(TSocket.java:178) ... 11 more The Apache Jenkins build system has built Hive-trunk-h0.21 (build #$BUILD_NUMBER) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1563/ to view the results.
[jira] [Commented] (HIVE-3225) NPE on a join query with authorization enabled
[ https://issues.apache.org/jira/browse/HIVE-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421222#comment-13421222 ] Hudson commented on HIVE-3225: -- Integrated in Hive-trunk-h0.21 #1563 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1563/]) HIVE-3225 : NPE on a join query with authorization enabled (Francis Liu via Ashutosh Chauhan) (Revision 1364853) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1364853 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java * /hive/trunk/ql/src/test/queries/clientnegative/join_nonexistent_part.q * /hive/trunk/ql/src/test/results/clientnegative/join_nonexistent_part.q.out NPE on a join query with authorization enabled -- Key: HIVE-3225 URL: https://issues.apache.org/jira/browse/HIVE-3225 Project: Hive Issue Type: Bug Components: Security Affects Versions: 0.9.0 Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.10.0 Attachments: HIVE-3225.patch when performing a join query which filters by a non-existent partition in the where clause (ie): select t1.a as a1, t2.a as a2 from t1 join t2 on t1.a=t2.a where t2.part=non-existent; It returns an NPE. It seems that the partition since non-existent is not part of the list of inputs (or maybe optimized out?). But the TableScanOperator still has a reference to it which causes an NPE after tableUsePartLevelAuth.get() returns null. FAILED: Hive Internal Error: java.lang.NullPointerException(null)java.lang.NullPointerExceptionat org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:617) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:486) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3286) Explicit skew join on user provided condition
[
https://issues.apache.org/jira/browse/HIVE-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3286:
Description:
Join operation on table with skewed data takes most of execution time handling
the skewed keys. But mostly we already know about that and even know what is
look like the skewed keys.
If we can explicitly assign reducer slots for the skewed keys, total execution
time could be greatly shortened.
As for a start, I've extended join grammar something like this.
{code}
select * from src a join src b on a.key=b.key skew on (a.key+1 50, a.key+1
100, a.key 150);
{code}
which means if above query is executed by 20 reducers, one reducer for a.key+1
50, one reducer for 50 = a.key+1 100, one reducer for 99 = a.key 150,
and 17 reducers for others (could be extended to assign more than one reducer
later)
This can be only used with common-inner-equi joins. And skew condition should
be composed of join keys only.
Work till done now will be updated shortly after code cleanup.
Skew expressions* in SKEW ON (expr, expr, ...) are evaluated sequentially at
runtime, and first 'true' one decides skew group for the row. Each skew group
has reserved partition slot(s), to which all rows in a group would be assigned.
The number of partition slot reserved for each group is decided also at runtime
by simple calculation of percentage. If a skew group is CLUSTER BY 20 PERCENT
and total partition slot (=number of reducer) is 20, that group will reserve 4
partition slots, etc.
DISTRIBUTE BY decides how the rows in a group is dispersed in the range of
reserved slots (If there is only one slot for a group, this is meaningless).
Currently, three distribution policies are available: RANDOM, KEYS,
expression.
1. RANDOM : rows of driver** alias are dispersed by random and rows of
non-driver alias are duplicated for all the slots (default if not specified)
2. KEYS : determined by hash value of keys (same with previous)
3. expression : determined by hash of object evaluated by user-provided
expression
Only possible with inner, equi, common-joins. Not yet supports join tree
merging.
Might be used by other RS users like SORT BY or GROUP BY
If there exists column statistics for the key, it could be possible to apply
automatically.
For example, if 20 reducers are used for the query below,
{code}
select count(*) from src a join src b on a.key=b.key skew on (
a.key = '0' CLUSTER BY 10 PERCENT,
b.key '100' CLUSTER BY 20 PERCENT DISTRIBUTE BY upper(b.key),
cast(a.key as int) 300 CLUSTER BY 40 PERCENT DISTRIBUTE BY KEYS);
{code}
group-0 will reserve slots 6~7, group-1 8~11, group-2 12~19 and others will
reserve slots 0~5.
For a row with key='0' from alias a, the row is randomly assigned in the range
of 6~7 (driver alias) : 6 or 7
For a row with key='0' from alias b, the row is disributed for all slots in 6~7
(non-driver alias) : 6 and 7
For a row with key='50', the row is assigned in the range of 8~11 by hashcode
of upper(b.key) : 8 + (hash(upper(key)) % 4)
For a row with key='500', the row is assigned in the range of 12~19 by hashcode
of join key : 12 + (hash(key) % 8)
For a row with key='200', this is not belong to any skew group : hash(key) % 6
*expressions in skew condition :
1. all expressions should be made of expression in join condition, which means
if join condition is a.key=b.key, user can make any expression with a.key
or b.key. But if join condition is a.key+1=b.key, user cannot make expression
with a.key solely (should make expression with a.key+1).
2. all expressions should reference one and only-one side of aliases. For
example, simple constant expressions or expressions referencing both side of
join condition (a.key+b.key100) is not allowed.
3. all functions in expression should be deteministic and stateless.
4. if DISTRIBUTED BY expression is used, distibution expression also should
have same alias with skew expression.
**driver alias :
1. driver alias means the sole referenced alias from skew expression, which is
important for RANDOM distribution. rows of driver alias are assigned to single
slot randomly, but rows of non-driver alias are duplicated for all the slots.
So, driver alias should be the biggest one in join aliases.
was:
Join operation on table with skewed data takes most of execution time handling
the skewed keys. But mostly we already know about that and even know what is
look like the skewed keys.
If we can explicitly assign reducer slots for the skewed keys, total execution
time could be greatly shortened.
As for a start, I've extended join grammar something like this.
{code}
select * from src a join src b on a.key=b.key skew on (a.key+1 50, a.key+1
100, a.key 150);
{code}
which means if above query is executed by 20 reducers, one reducer for a.key+1
50, one reducer for 50 = a.key+1 100, one reducer for 99 = a.key
[jira] [Updated] (HIVE-3219) BucketizedHiveInputFormat should be automatically used with SMBJoin
[
https://issues.apache.org/jira/browse/HIVE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3219:
Status: Patch Available (was: Open)
Addressed comments
BucketizedHiveInputFormat should be automatically used with SMBJoin
---
Key: HIVE-3219
URL: https://issues.apache.org/jira/browse/HIVE-3219
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.10.0
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3219.1.patch.txt
For SMBJoin, BucketizedHiveInputFormat should(could?) be used with set
command.
{noformat}
set hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
{noformat}
But this affects all of the MR taks in a query, which might not be intended.
With some configuration, it should be automatically configured only for MR
tasks which contains SMBJoin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3268) expressions in cluster by are not working
[ https://issues.apache.org/jira/browse/HIVE-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3268: - Status: Open (was: Patch Available) Some tests failed - debugging expressions in cluster by are not working - Key: HIVE-3268 URL: https://issues.apache.org/jira/browse/HIVE-3268 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain The following query fails: select key+key, value from src cluster by key+key, value; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3268) expressions in cluster by are not working
[ https://issues.apache.org/jira/browse/HIVE-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3268: - Status: Patch Available (was: Open) expressions in cluster by are not working - Key: HIVE-3268 URL: https://issues.apache.org/jira/browse/HIVE-3268 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain The following query fails: select key+key, value from src cluster by key+key, value; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3292) Table Statistics
Namit Jain created HIVE-3292: Summary: Table Statistics Key: HIVE-3292 URL: https://issues.apache.org/jira/browse/HIVE-3292 Project: Hive Issue Type: Bug Reporter: Namit Jain There is a problem with table statistics: If 2 partitions of a table are being created/updated concurrently, table level stats may not be correct. For eg. if a table T has 2 partitions P1 and P2 with number of rows 10 and 20 respectively, and both the partitions are being updated concurrently (via insert overwrite ...), the table object is obtained in StatsTask and updated. Even with concurrency turned on, the table is not locked, and the two table statistics updates may lead to one to them being lost. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3293) Load file into a table does not update table statistics
Namit Jain created HIVE-3293: Summary: Load file into a table does not update table statistics Key: HIVE-3293 URL: https://issues.apache.org/jira/browse/HIVE-3293 Project: Hive Issue Type: Bug Reporter: Namit Jain create table smb_bucket_1(key int, value string); load data local inpath '../data/files/smbbucket_1.rc' overwrite into table smb_bucket_1; load data local inpath '../data/files/smbbucket_2.rc' overwrite into table smb_bucket_2; does not update the stats for smb_bucket_1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3294) alter table add partition does not update the statistics of the partition or the table
Namit Jain created HIVE-3294: Summary: alter table add partition does not update the statistics of the partition or the table Key: HIVE-3294 URL: https://issues.apache.org/jira/browse/HIVE-3294 Project: Hive Issue Type: Bug Reporter: Namit Jain This may be specially bad for external tables, whose partitions are being created via alter table add partition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3219) BucketizedHiveInputFormat should be automatically used with SMBJoin
[
https://issues.apache.org/jira/browse/HIVE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-3219:
-
Status: Open (was: Patch Available)
BucketizedHiveInputFormat should be automatically used with SMBJoin
---
Key: HIVE-3219
URL: https://issues.apache.org/jira/browse/HIVE-3219
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.10.0
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3219.1.patch.txt
For SMBJoin, BucketizedHiveInputFormat should(could?) be used with set
command.
{noformat}
set hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
{noformat}
But this affects all of the MR taks in a query, which might not be intended.
With some configuration, it should be automatically configured only for MR
tasks which contains SMBJoin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #85
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/ -- [...truncated 10115 lines...] [echo] Project: odbc [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/odbc/src/conf does not exist. ivy-resolve-test: [echo] Project: odbc ivy-retrieve-test: [echo] Project: odbc compile-test: [echo] Project: odbc create-dirs: [echo] Project: serde [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/serde/src/test/resources does not exist. init: [echo] Project: serde ivy-init-settings: [echo] Project: serde ivy-resolve: [echo] Project: serde [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-serde-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/ivy/report/org.apache.hive-hive-serde-default.html ivy-retrieve: [echo] Project: serde dynamic-serde: compile: [echo] Project: serde ivy-resolve-test: [echo] Project: serde ivy-retrieve-test: [echo] Project: serde compile-test: [echo] Project: serde [javac] Compiling 26 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/serde/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. create-dirs: [echo] Project: service [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/service/src/test/resources does not exist. init: [echo] Project: service ivy-init-settings: [echo] Project: service ivy-resolve: [echo] Project: service [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-service-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/ivy/report/org.apache.hive-hive-service-default.html ivy-retrieve: [echo] Project: service compile: [echo] Project: service ivy-resolve-test: [echo] Project: service ivy-retrieve-test: [echo] Project: service compile-test: [echo] Project: service [javac] Compiling 2 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/service/test/classes test: [echo] Project: hive test-shims: [echo] Project: hive test-conditions: [echo] Project: shims gen-test: [echo] Project: shims create-dirs: [echo] Project: shims [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/test/resources does not exist. init: [echo] Project: shims ivy-init-settings: [echo] Project: shims ivy-resolve: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-shims-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/ivy/report/org.apache.hive-hive-shims-default.html ivy-retrieve: [echo] Project: shims compile: [echo] Project: shims [echo] Building shims 0.20 build_shims: [echo] Project: shims [echo] Compiling https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.20/java against hadoop 0.20.2 (https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/85/artifact/hive/build/hadoopcore/hadoop-0.20.2) ivy-init-settings: [echo] Project: shims ivy-resolve-hadoop-shim: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml ivy-retrieve-hadoop-shim: [echo] Project: shims [echo] Building shims 0.20S build_shims: [echo] Project: shims [echo] Compiling
[jira] [Commented] (HIVE-3126) Generate build the velocity based Hive tests on windows by fixing the path issues
[ https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421569#comment-13421569 ] Kanna Karanam commented on HIVE-3126: - @Carl, @Ashutosh - can you please review the updated diff? Thanks. Generate build the velocity based Hive tests on windows by fixing the path issues --- Key: HIVE-3126 URL: https://issues.apache.org/jira/browse/HIVE-3126 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0, 0.10.0, 0.9.1 Reporter: Kanna Karanam Assignee: Kanna Karanam Labels: Windows, test Fix For: 0.10.0 Attachments: HIVE-3126.1.patch.txt, HIVE-3126.10.patch.txt, HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, HIVE-3126.8.patch.txt, HIVE-3126.9.patch.txt 1)Escape the backward slash in Canonical Path if unit test runs on windows. 2)Diff comparison – a. Ignore the extra spacing on windows b. Ignore the different line endings on windows Unix c. Convert the file paths to windows specific. (Handle spaces etc..) 3)Set the right file scheme class path separators while invoking the junit task from -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2909: Resolution: Fixed Fix Version/s: 0.10.0 Status: Resolved (was: Patch Available) Committed, thanks Namit. SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Namit Jain Priority: Minor Fix For: 0.10.0 Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch, HIVE-2909.D2595.5.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3279) Table schema not being copied to Partitions with no columns
[
https://issues.apache.org/jira/browse/HIVE-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421823#comment-13421823
]
Travis Crawford commented on HIVE-3279:
---
Hey [~ashutoshc] - fixing this issue is pretty high priority for me, any
thoughts on next steps?
Table schema not being copied to Partitions with no columns
---
Key: HIVE-3279
URL: https://issues.apache.org/jira/browse/HIVE-3279
Project: Hive
Issue Type: Bug
Reporter: Travis Crawford
Assignee: Travis Crawford
Hive has a feature where {{Partition}}'s without any defined columns use the
{{Table}} schema. This happens in
{{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167]}}
{code}
// set default if columns are not set
if (tPartition.getSd().getCols() == null) {
if (table.getCols() != null) {
tPartition.getSd().setCols(table.getCols());
}
}
{code}
There's an issue though, because
{{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
initializes cols to an empty array, which of course is not null, causing the
above feature to not work as expected.
I'm not sure of the fix - is there a case where cols can indeed be null? I
think the best thing to do here is:
{code}
-if (tPartition.getSd().getCols() == null) {
+if (tPartition.getSd().getCols() == null ||
tPartition.getSd().getCols().size() == 0) {
{code}
Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3126) Generate build the velocity based Hive tests on windows by fixing the path issues
[ https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421853#comment-13421853 ] Ashutosh Chauhan commented on HIVE-3126: @Kanna: Thanks, Kanna. I will take a look. Generate build the velocity based Hive tests on windows by fixing the path issues --- Key: HIVE-3126 URL: https://issues.apache.org/jira/browse/HIVE-3126 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0, 0.10.0, 0.9.1 Reporter: Kanna Karanam Assignee: Kanna Karanam Labels: Windows, test Fix For: 0.10.0 Attachments: HIVE-3126.1.patch.txt, HIVE-3126.10.patch.txt, HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, HIVE-3126.8.patch.txt, HIVE-3126.9.patch.txt 1)Escape the backward slash in Canonical Path if unit test runs on windows. 2)Diff comparison – a. Ignore the extra spacing on windows b. Ignore the different line endings on windows Unix c. Convert the file paths to windows specific. (Handle spaces etc..) 3)Set the right file scheme class path separators while invoking the junit task from -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive CLI wikidoc review
The Hive
CLIhttps://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli
wikidoc
differs slightly from the xdocs
versionhttp://hive.apache.org/docs/r0.9.0/language_manual/cli.html
(source-controlled
xml file) and they both need updating, so I merged them into a single
wikidoc for review by this community. You can find the draft here: Revised
LanguageManual
CLIhttps://cwiki.apache.org/confluence/display/Hive/Revised+LanguageManual+Cli
.
After the wikidoc has been reviewed and fixed, I'll apply the same changes
to the xdocs version. Eventually only one copy should remain, but that's a
separate issue. I'll be revising more wikidocs as time permits; the CLI
doc is first because diverging copies can cause trouble.
Besides a simple merge, this draft includes technical changes based on
Thejas Nair's review comments as well as some editorial changes. Please
let me know (or edit the draft) if you find any errors or want any changes.
If more information is available, please provide it.
Q1: In the Hive Resources section is the syntax for DELETE FILE | JAR |
ARCHIVE correct?
DELETE { FILE[S] | JAR[S] | ARCHIVE[S] } [filepath1 filepath2 ..]
Q2: If DELETE does not require a filepath, what's the result – deleting *
all* files | jars | archives?
– Lefty Leverenz, *technical writer on a mission to improve Hive doc*s
[jira] [Created] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
Kevin Wilfong created HIVE-3295: --- Summary: HIVE-3128 introduced bug causing dynamic partitioning to fail Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Kevin Wilfong HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3295: Priority: Blocker (was: Major) HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3295: Component/s: Query Processor Affects Version/s: 0.10.0 HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421883#comment-13421883 ] Kevin Wilfong commented on HIVE-3295: - https://reviews.facebook.net/D4311 HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3295: Status: Patch Available (was: Open) HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3295: Attachment: HIVE-3295.1.patch.txt HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker Attachments: HIVE-3295.1.patch.txt HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3126) Generate build the velocity based Hive tests on windows by fixing the path issues
[ https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421895#comment-13421895 ] Carl Steinbach commented on HIVE-3126: -- Looks good to me. Kanna, thanks for making all of these changes. Generate build the velocity based Hive tests on windows by fixing the path issues --- Key: HIVE-3126 URL: https://issues.apache.org/jira/browse/HIVE-3126 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0, 0.10.0, 0.9.1 Reporter: Kanna Karanam Assignee: Kanna Karanam Labels: Windows, test Fix For: 0.10.0 Attachments: HIVE-3126.1.patch.txt, HIVE-3126.10.patch.txt, HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, HIVE-3126.8.patch.txt, HIVE-3126.9.patch.txt 1)Escape the backward slash in Canonical Path if unit test runs on windows. 2)Diff comparison – a. Ignore the extra spacing on windows b. Ignore the different line endings on windows Unix c. Convert the file paths to windows specific. (Handle spaces etc..) 3)Set the right file scheme class path separators while invoking the junit task from -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421898#comment-13421898 ] Hudson commented on HIVE-2909: -- Integrated in Hive-trunk-h0.21 #1565 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1565/]) HIVE-2909. SHOW COLUMNS table_name; to provide a comma-delimited list of columns. (njain via kevinwilfong) (Revision 1365222) Result = SUCCESS kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1365222 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowColumnsDesc.java * /hive/trunk/ql/src/test/queries/clientnegative/show_columns1.q * /hive/trunk/ql/src/test/queries/clientnegative/show_columns2.q * /hive/trunk/ql/src/test/queries/clientnegative/show_columns3.q * /hive/trunk/ql/src/test/queries/clientpositive/show_columns.q * /hive/trunk/ql/src/test/results/clientnegative/show_columns1.q.out * /hive/trunk/ql/src/test/results/clientnegative/show_columns2.q.out * /hive/trunk/ql/src/test/results/clientnegative/show_columns3.q.out * /hive/trunk/ql/src/test/results/clientpositive/show_columns.q.out SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Namit Jain Priority: Minor Fix For: 0.10.0 Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch, HIVE-2909.D2595.5.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1565 - Fixed
Changes for Build #1561 Changes for Build #1562 [kevinwilfong] HIVE-3261. alter the number of buckets for a non-empty partitioned table should not be allowed. (njain via kevinwilfong) Changes for Build #1563 [hashutosh] HIVE-3225 : NPE on a join query with authorization enabled (Francis Liu via Ashutosh Chauhan) Changes for Build #1564 Changes for Build #1565 [kevinwilfong] HIVE-2909. SHOW COLUMNS table_name; to provide a comma-delimited list of columns. (njain via kevinwilfong) All tests passed The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1565) Status: Fixed Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1565/ to view the results.
[jira] [Updated] (HIVE-3249) Upgrade guava to 11.0.2
[ https://issues.apache.org/jira/browse/HIVE-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3249: - Resolution: Fixed Fix Version/s: 0.10.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Zhenxiao! Upgrade guava to 11.0.2 --- Key: HIVE-3249 URL: https://issues.apache.org/jira/browse/HIVE-3249 Project: Hive Issue Type: Bug Components: Build Infrastructure Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Fix For: 0.10.0 Attachments: HIVE-3249.1.patch.txt Hadoop has upgraded to this new version of Guava. We should, too, so we don't have compatibility issues running on Hadoop 2.0+ currently, hive is using guava-r09.jar we should update it to guava-11.0.2.jar -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3126) Generate build the velocity based Hive tests on windows by fixing the path issues
[ https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421913#comment-13421913 ] Ashutosh Chauhan commented on HIVE-3126: +1 Will commit if tests pass. Generate build the velocity based Hive tests on windows by fixing the path issues --- Key: HIVE-3126 URL: https://issues.apache.org/jira/browse/HIVE-3126 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0, 0.10.0, 0.9.1 Reporter: Kanna Karanam Assignee: Kanna Karanam Labels: Windows, test Fix For: 0.10.0 Attachments: HIVE-3126.1.patch.txt, HIVE-3126.10.patch.txt, HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, HIVE-3126.8.patch.txt, HIVE-3126.9.patch.txt 1)Escape the backward slash in Canonical Path if unit test runs on windows. 2)Diff comparison – a. Ignore the extra spacing on windows b. Ignore the different line endings on windows Unix c. Convert the file paths to windows specific. (Handle spaces etc..) 3)Set the right file scheme class path separators while invoking the junit task from -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421919#comment-13421919 ] Ashutosh Chauhan commented on HIVE-3295: +1. Your solution of moving out that method from FileUtils to CompressionUtils is a good one. We definitely don't want to send unnecessary jars to cluster. Please commit if tests pass. HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker Attachments: HIVE-3295.1.patch.txt HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3295: - Status: Open (was: Patch Available) @Kevin: The new file is missing an ASF header. HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker Attachments: HIVE-3295.1.patch.txt HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Hive CLI wikidoc review
The xdoc should be removed its dead and out of sync.
On Tue, Jul 24, 2012 at 7:21 PM, Lefty Leverenz le...@hortonworks.com wrote:
The Hive
CLIhttps://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli
wikidoc
differs slightly from the xdocs
versionhttp://hive.apache.org/docs/r0.9.0/language_manual/cli.html
(source-controlled
xml file) and they both need updating, so I merged them into a single
wikidoc for review by this community. You can find the draft here: Revised
LanguageManual
CLIhttps://cwiki.apache.org/confluence/display/Hive/Revised+LanguageManual+Cli
.
After the wikidoc has been reviewed and fixed, I'll apply the same changes
to the xdocs version. Eventually only one copy should remain, but that's a
separate issue. I'll be revising more wikidocs as time permits; the CLI
doc is first because diverging copies can cause trouble.
Besides a simple merge, this draft includes technical changes based on
Thejas Nair's review comments as well as some editorial changes. Please
let me know (or edit the draft) if you find any errors or want any changes.
If more information is available, please provide it.
Q1: In the Hive Resources section is the syntax for DELETE FILE | JAR |
ARCHIVE correct?
DELETE { FILE[S] | JAR[S] | ARCHIVE[S] } [filepath1 filepath2 ..]
Q2: If DELETE does not require a filepath, what's the result – deleting *
all* files | jars | archives?
– Lefty Leverenz, *technical writer on a mission to improve Hive doc*s
Re: Hive CLI wikidoc review
Okay, we can get rid of the xdoc but what about this information from it?
The cli when invoked without the -i option will attempt to load
HIVE_HOME/bin/.hiverc and $HOME/.hiverc as initialization files.
Is it still useful, or should I delete it from the draft?
– Lefty
On Tue, Jul 24, 2012 at 6:44 PM, Edward Capriolo edlinuxg...@gmail.comwrote:
The xdoc should be removed its dead and out of sync.
On Tue, Jul 24, 2012 at 7:21 PM, Lefty Leverenz le...@hortonworks.com
wrote:
The Hive CLI
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli
wikidoc
differs slightly from the xdocs
versionhttp://hive.apache.org/docs/r0.9.0/language_manual/cli.html
(source-controlled
xml file) and they both need updating, so I merged them into a single
wikidoc for review by this community. You can find the draft here:
Revised
LanguageManual CLI
https://cwiki.apache.org/confluence/display/Hive/Revised+LanguageManual+Cli
.
After the wikidoc has been reviewed and fixed, I'll apply the same
changes
to the xdocs version. Eventually only one copy should remain, but
that's a
separate issue. I'll be revising more wikidocs as time permits; the CLI
doc is first because diverging copies can cause trouble.
Besides a simple merge, this draft includes technical changes based on
Thejas Nair's review comments as well as some editorial changes. Please
let me know (or edit the draft) if you find any errors or want any
changes.
If more information is available, please provide it.
Q1: In the Hive Resources section is the syntax for DELETE FILE | JAR
|
ARCHIVE correct?
DELETE { FILE[S] | JAR[S] | ARCHIVE[S] } [filepath1 filepath2 ..]
Q2: If DELETE does not require a filepath, what's the result – deleting
*
all* files | jars | archives?
– Lefty Leverenz, *technical writer on a mission to improve Hive doc*s
[jira] [Commented] (HIVE-3126) Generate build the velocity based Hive tests on windows by fixing the path issues
[ https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421947#comment-13421947 ] Kanna Karanam commented on HIVE-3126: - Thanks Ashutosh, Carl. This JIRA helps developers to build, develop , debug hive on windows. I am not done yet until I fix all other unit test failures but it helps the community to play on windows. Generate build the velocity based Hive tests on windows by fixing the path issues --- Key: HIVE-3126 URL: https://issues.apache.org/jira/browse/HIVE-3126 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0, 0.10.0, 0.9.1 Reporter: Kanna Karanam Assignee: Kanna Karanam Labels: Windows, test Fix For: 0.10.0 Attachments: HIVE-3126.1.patch.txt, HIVE-3126.10.patch.txt, HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, HIVE-3126.8.patch.txt, HIVE-3126.9.patch.txt 1)Escape the backward slash in Canonical Path if unit test runs on windows. 2)Diff comparison – a. Ignore the extra spacing on windows b. Ignore the different line endings on windows Unix c. Convert the file paths to windows specific. (Handle spaces etc..) 3)Set the right file scheme class path separators while invoking the junit task from -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2608) Do not require AS a,b,c part in LATERAL VIEW
[ https://issues.apache.org/jira/browse/HIVE-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis reassigned HIVE-2608: --- Assignee: Navis Do not require AS a,b,c part in LATERAL VIEW Key: HIVE-2608 URL: https://issues.apache.org/jira/browse/HIVE-2608 Project: Hive Issue Type: Improvement Components: Query Processor, UDF Reporter: Igor Kabiljo Assignee: Navis Priority: Minor Currently, it is required to state column names when LATERAL VIEW is used. That shouldn't be necessary, since UDTF returns struct which contains column names - and they should be used by default. For example, it would be great if this was possible: SELECT t.*, t.key1 + t.key4 FROM some_table LATERAL VIEW JSON_TUPLE(json, 'key1', 'key2', 'key3', 'key3') t; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2608) Do not require AS a,b,c part in LATERAL VIEW
[ https://issues.apache.org/jira/browse/HIVE-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-2608: Affects Version/s: 0.10.0 Status: Patch Available (was: Open) https://reviews.facebook.net/D4317 Do not require AS a,b,c part in LATERAL VIEW Key: HIVE-2608 URL: https://issues.apache.org/jira/browse/HIVE-2608 Project: Hive Issue Type: Improvement Components: Query Processor, UDF Affects Versions: 0.10.0 Reporter: Igor Kabiljo Assignee: Navis Priority: Minor Currently, it is required to state column names when LATERAL VIEW is used. That shouldn't be necessary, since UDTF returns struct which contains column names - and they should be used by default. For example, it would be great if this was possible: SELECT t.*, t.key1 + t.key4 FROM some_table LATERAL VIEW JSON_TUPLE(json, 'key1', 'key2', 'key3', 'key3') t; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1450) always catch exception when invoke executeUpdate in jdbc
[
https://issues.apache.org/jira/browse/HIVE-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-1450:
-
Status: Open (was: Patch Available)
Comments on phabricator
always catch exception when invoke executeUpdate in jdbc
Key: HIVE-1450
URL: https://issues.apache.org/jira/browse/HIVE-1450
Project: Hive
Issue Type: Bug
Components: JDBC
Reporter: Alexey Diomin
Assignee: Carl Steinbach
Attachments: HIVE-1450.D4293.1.patch
Request executed in hive, but always return exception
error in ./jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java
{code}
public int executeUpdate(String sql) throws SQLException {
try {
client.execute(sql);
} catch (Exception ex) {
throw new SQLException(ex.toString());
}
throw new SQLException(Method not supported);
}
{code}
executeQuery work correct
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3296) comments before a set statement do not work
[ https://issues.apache.org/jira/browse/HIVE-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3296: - Summary: comments before a set statement do not work (was: comments before a set statement does no work) comments before a set statement do not work --- Key: HIVE-3296 URL: https://issues.apache.org/jira/browse/HIVE-3296 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Namit Jain The following test fails::: -- test comment set hive.test=foo; create table tst(key string); The output is:: FAILED: ParseException line 2:0 cannot recognize input near 'set' 'hive' '.' However, if the comment position is changed it works fine. The following test: -- test comment create table tst(key string); set hive.test=foo; works fine -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3296) comments before a set statement in a test do not work
[ https://issues.apache.org/jira/browse/HIVE-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3296: - Component/s: Testing Infrastructure comments before a set statement in a test do not work - Key: HIVE-3296 URL: https://issues.apache.org/jira/browse/HIVE-3296 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Namit Jain The following test fails::: -- test comment set hive.test=foo; create table tst(key string); The output is:: FAILED: ParseException line 2:0 cannot recognize input near 'set' 'hive' '.' However, if the comment position is changed it works fine. The following test: -- test comment create table tst(key string); set hive.test=foo; works fine -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3296) comments before a set statement does no work
Namit Jain created HIVE-3296: Summary: comments before a set statement does no work Key: HIVE-3296 URL: https://issues.apache.org/jira/browse/HIVE-3296 Project: Hive Issue Type: Bug Reporter: Namit Jain The following test fails::: -- test comment set hive.test=foo; create table tst(key string); The output is:: FAILED: ParseException line 2:0 cannot recognize input near 'set' 'hive' '.' However, if the comment position is changed it works fine. The following test: -- test comment create table tst(key string); set hive.test=foo; works fine -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3297) change hive.auto.convert.join's default value to true
Namit Jain created HIVE-3297: Summary: change hive.auto.convert.join's default value to true Key: HIVE-3297 URL: https://issues.apache.org/jira/browse/HIVE-3297 Project: Hive Issue Type: Bug Reporter: Namit Jain For unit tests also, this parameter should be set to true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3295: Status: Patch Available (was: Open) HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker Attachments: HIVE-3295.1.patch.txt HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422021#comment-13422021 ] Kevin Wilfong commented on HIVE-3295: - Added the header. HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker Attachments: HIVE-3295.1.patch.txt HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail
[ https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3295: Attachment: HIVE-3295.2.patch.txt HIVE-3128 introduced bug causing dynamic partitioning to fail - Key: HIVE-3295 URL: https://issues.apache.org/jira/browse/HIVE-3295 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Blocker Attachments: HIVE-3295.1.patch.txt, HIVE-3295.2.patch.txt HIVE-3128 introduced a new commons-compress jar and imports classes from it in FileUtils.java The FileUtils class is accessed by dynamic partitioning in the map reduce cluster where the jar is not available, causing the query to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
