Jenkins build is back to normal : Hive-0.10.0-SNAPSHOT-h0.20.1 #131
See https://builds.apache.org/job/Hive-0.10.0-SNAPSHOT-h0.20.1/131/
[jira] [Commented] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642699#comment-13642699 ] Namit Jain commented on HIVE-4409: -- my bad, will do Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Fix For: 0.12.0 Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, HIVE-4409.D10539.2.patch If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
Namit Jain created HIVE-4424: Summary: MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
[ https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642700#comment-13642700 ] Namit Jain commented on HIVE-4424: -- Deleted metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 --- Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
[ https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4424: - Status: Patch Available (was: Open) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 --- Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
[ https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4424: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 --- Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4232) JDBC2 HiveConnection has odd defaults
[ https://issues.apache.org/jira/browse/HIVE-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Drome updated HIVE-4232: -- Attachment: HIVE-4232-2.patch WIP patch incorporating the comments. NOSASL/NONE transport layer test failing with latest trunk code. JDBC2 HiveConnection has odd defaults - Key: HIVE-4232 URL: https://issues.apache.org/jira/browse/HIVE-4232 Project: Hive Issue Type: Bug Components: HiveServer2, JDBC Affects Versions: 0.11.0 Reporter: Chris Drome Assignee: Chris Drome Fix For: 0.11.0 Attachments: HIVE-4232-1.patch, HIVE-4232-2.patch, HIVE-4232.patch HiveConnection defaults to using a plain SASL transport if auth is not set. To get a raw transport auth must be set to noSasl; furthermore noSasl is case sensitive. Code tries to infer Kerberos or plain authentication based on the presence of principal. There is no provision for specifying QOP level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4423) Improve RCFile::sync(long) 10x
[ https://issues.apache.org/jira/browse/HIVE-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-4423: -- Attachment: HIVE-4423.patch buffer RCFile::sync reads into io.bytes.per.checksum chunks Improve RCFile::sync(long) 10x -- Key: HIVE-4423 URL: https://issues.apache.org/jira/browse/HIVE-4423 Project: Hive Issue Type: Improvement Environment: Ubuntu LXC (1 SSD, 1 disk, 32 gigs of RAM) Reporter: Gopal V Assignee: Gopal V Priority: Minor Fix For: 0.11.0 Attachments: HIVE-4423.patch RCFile::sync(long) takes approx ~1 second everytime it gets called because of the inner loops in the function. From what was observed with HDFS-4710, single byte reads are an order of magnitude slower than larger 512 byte buffer reads. Even when disk I/O is buffered to this size, there is overhead due to the synchronized read() methods in BlockReaderLocal RemoteBlockReader classes. Removing the readByte() calls in RCFile.sync(long) with a readFully(512 byte) call will speed this function 10x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4423) Improve RCFile::sync(long) 10x
[ https://issues.apache.org/jira/browse/HIVE-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642745#comment-13642745 ] Gopal V commented on HIVE-4423: --- || split location || before || after || | store_sales/00_0:67108864+67108864 | 748 ms |81 ms | | store_sales/02_0:67108864+67108864 | 966 ms |54 ms | | store_sales/04_0:67108864+67108864 | 948 ms |51 ms | | store_sales/06_0:67108864+67108864 | 922 ms |42 ms | | store_sales/08_0:67108864+67108864 | 842 ms |40 ms | | store_sales/10_0:67108864+67108864 | 1302 ms | 82 ms | | store_sales/12_0:67108864+67108864 | 989 ms |50 ms | | store_sales/14_0:67108864+67108864 | 970 ms |43 ms | | store_sales/01_0:67108864+67108864 | 829 ms |47 ms | | store_sales/03_0:67108864+67108864 | 811 ms |43 ms | | store_sales/07_0:67108864+67108864 | 865 ms |51 ms | | store_sales/05_0:67108864+67108864 | 1042 ms | 59 ms | | store_sales/09_0:67108864+67108864 | 902 ms |39 ms | | store_sales/11_0:67108864+67108864 | 1046 ms | 42 ms | | store_sales/13_0:67108864+67108864 | 1048 ms | 44 ms | As expected, the function is faster by an order of magnitude fast enough to not need more optimization in the inner sync.length for loop. Over all, the query was faster by 2+ seconds for a 28 second query (since we have 8 slots and 15 mappers, so that's expected). Improve RCFile::sync(long) 10x -- Key: HIVE-4423 URL: https://issues.apache.org/jira/browse/HIVE-4423 Project: Hive Issue Type: Improvement Environment: Ubuntu LXC (1 SSD, 1 disk, 32 gigs of RAM) Reporter: Gopal V Assignee: Gopal V Priority: Minor Fix For: 0.11.0 Attachments: HIVE-4423.patch RCFile::sync(long) takes approx ~1 second everytime it gets called because of the inner loops in the function. From what was observed with HDFS-4710, single byte reads are an order of magnitude slower than larger 512 byte buffer reads. Even when disk I/O is buffered to this size, there is overhead due to the synchronized read() methods in BlockReaderLocal RemoteBlockReader classes. Removing the readByte() calls in RCFile.sync(long) with a readFully(512 byte) call will speed this function 10x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21 #358
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/358/ -- [...truncated 36527 lines...] [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/jenkins/hive_2013-04-26_04-11-39_712_7484477311902458318/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/358/artifact/hive/build/service/tmp/hive_job_log_jenkins_201304260411_2137534265.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: DROPTABLE [junit] Copying file: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Output: default@testhivedrivertable [junit] Copying data from https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt [junit] Loading data to table default.testhivedrivertable [junit] POSTHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: select * from testhivedrivertable limit 10 [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/jenkins/hive_2013-04-26_04-11-44_225_1771408511806018940/-mr-1 [junit] POSTHOOK: query: select * from testhivedrivertable limit 10 [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/jenkins/hive_2013-04-26_04-11-44_225_1771408511806018940/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/358/artifact/hive/build/service/tmp/hive_job_log_jenkins_201304260411_881202622.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/358/artifact/hive/build/service/tmp/hive_job_log_jenkins_201304260411_818595256.txt [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/358/artifact/hive/build/service/tmp/hive_job_log_jenkins_201304260411_2055466638.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (key int, value
Review Request: Improve RCFile::sync(long) by 10x
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10795/ --- Review request for hive, Ashutosh Chauhan and Gunther Hagleitner. Description --- Speed up RCFile::sync() by reading large blocks of data from HDFS rather than using readByte() on the input stream. This improves the loop behaviour and reduces the number of calls on the synchronized read() methods within HDFS, resulting in a 10x performance boost to this function. In real time, it converts a call that takes upto a second and brings it below 100ms, by reading 512 byte chunks instead of reading data 1 byte at a time. This addresses bug HIVE-4423. https://issues.apache.org/jira/browse/HIVE-4423 Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java d3d98d0 Diff: https://reviews.apache.org/r/10795/diff/ Testing --- ant test -Dtestcase=TestRCFile -Dmodule=ql ant test -Dtestcase=TestCliDriver -Dqfile_regex=.*rcfile.* -Dmodule=ql And benchmarking with count(1) on the store_sales rcfile table at scale=10 before: 43.8, after: 39.5 Thanks, Gopal V
[jira] [Commented] (HIVE-4005) Column truncation
[ https://issues.apache.org/jira/browse/HIVE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642866#comment-13642866 ] Hudson commented on HIVE-4005: -- Integrated in Hive-trunk-hadoop2 #175 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/175/]) HIVE-4005 Column truncation (Kevin Wilfong via namit) (Revision 1475662) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1475662 Files : * /hive/trunk/build-common.xml * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileKeyBufferWrapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileValueBufferWrapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateMapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateWork.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java * /hive/trunk/ql/src/test/queries/clientnegative/truncate_bucketed_column.q * /hive/trunk/ql/src/test/queries/clientnegative/truncate_column_indexed_table.q * /hive/trunk/ql/src/test/queries/clientnegative/truncate_column_list_bucketing.q * /hive/trunk/ql/src/test/queries/clientnegative/truncate_column_seqfile.q * /hive/trunk/ql/src/test/queries/clientnegative/truncate_nonexistant_column.q * /hive/trunk/ql/src/test/queries/clientnegative/truncate_partition_column.q * /hive/trunk/ql/src/test/queries/clientnegative/truncate_partition_column2.q * /hive/trunk/ql/src/test/queries/clientpositive/truncate_column.q * /hive/trunk/ql/src/test/queries/clientpositive/truncate_column_buckets.q * /hive/trunk/ql/src/test/queries/clientpositive/truncate_column_list_bucket.q * /hive/trunk/ql/src/test/queries/clientpositive/truncate_column_merge.q * /hive/trunk/ql/src/test/results/clientnegative/truncate_bucketed_column.q.out * /hive/trunk/ql/src/test/results/clientnegative/truncate_column_archived.q.out * /hive/trunk/ql/src/test/results/clientnegative/truncate_column_indexed_table.q.out * /hive/trunk/ql/src/test/results/clientnegative/truncate_column_list_bucketing.q.out * /hive/trunk/ql/src/test/results/clientnegative/truncate_column_seqfile.q.out * /hive/trunk/ql/src/test/results/clientnegative/truncate_nonexistant_column.q.out * /hive/trunk/ql/src/test/results/clientnegative/truncate_partition_column.q.out * /hive/trunk/ql/src/test/results/clientnegative/truncate_partition_column2.q.out * /hive/trunk/ql/src/test/results/clientpositive/truncate_column.q.out * /hive/trunk/ql/src/test/results/clientpositive/truncate_column_buckets.q.out * /hive/trunk/ql/src/test/results/clientpositive/truncate_column_list_bucket.q.out * /hive/trunk/ql/src/test/results/clientpositive/truncate_column_merge.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/BytesRefArrayWritable.java Column truncation - Key: HIVE-4005 URL: https://issues.apache.org/jira/browse/HIVE-4005 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.12.0 Attachments: HIVE-4005.1.patch.txt, HIVE-4005.2.patch.txt, HIVE-4005.3.patch.txt, HIVE-4005.4.patch.txt, HIVE-4005.5.patch.txt, HIVE-4005.6.patch.txt, HIVE-4005.6.patch.txt, HIVE-4005.7.patch.txt Column truncation allows users to remove data for columns that are no longer useful. This is done by removing the data for the column and setting the length of the column data and related lengths to 0 in the RC file header. RC file was fixed to recognize columns with lengths of zero to be empty and are treated as if the column doesn't exist in the data, a null is returned for every value of that column in every row. This is the same thing that happens when more columns are selected than exist in the file. A new command was added to the CLI TRUNCATE TABLE ... PARTITION ... COLUMNS ... This launches a map only job where each mapper rewrites a single file without the unnecessary column data and the adjusted headers. It does not uncompress/deserialize the data so it is much faster than rewriting the data with NULLs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please
[jira] [Commented] (HIVE-4412) PTFDesc tries serialize transient fields like OIs, etc.
[ https://issues.apache.org/jira/browse/HIVE-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642865#comment-13642865 ] Hudson commented on HIVE-4412: -- Integrated in Hive-trunk-hadoop2 #175 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/175/]) HIVE-4412 : PTFDesc tries serialize transient fields like OIs, etc. (Navis via Ashutosh Chauhan) (Revision 1475642) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1475642 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLeadLag.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionEvaluator.java PTFDesc tries serialize transient fields like OIs, etc. --- Key: HIVE-4412 URL: https://issues.apache.org/jira/browse/HIVE-4412 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.12.0 Attachments: HIVE-4412.D10515.1.patch windowing_expression.q shows warning messages like this. {noformat} java.lang.InstantiationException: org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableConstantDoubleObjectInspector Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... java.lang.InstantiationException: org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... java.lang.IllegalAccessException: Class sun.reflect.misc.Trampoline can not access a member of class org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableDoubleObjectInspector with modifiers Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... {noformat} hive is serializing temporary OIs in it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4358) Check for Map side processing in PTFOp is no longer valid
[ https://issues.apache.org/jira/browse/HIVE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642864#comment-13642864 ] Hudson commented on HIVE-4358: -- Integrated in Hive-trunk-hadoop2 #175 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/175/]) HIVE-4358 : Check for Map side processing in PTFOp is no longer valid (Harish Butani via Ashutosh Chauhan) (Revision 1475880) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1475880 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java * /hive/trunk/ql/src/test/queries/clientpositive/windowing.q * /hive/trunk/ql/src/test/results/clientpositive/windowing.q.out Check for Map side processing in PTFOp is no longer valid - Key: HIVE-4358 URL: https://issues.apache.org/jira/browse/HIVE-4358 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Fix For: 0.12.0 Attachments: HIVE-4358.D10473.1.patch With the changes for ReduceSinkDedup it is no longer true that a non Map-side PTF Operator is preceded by an ExtractOp. For e.g. following query can produce the issue: {noformat} create view IF NOT EXISTS mfgr_price_view as select p_mfgr, p_brand, sum(p_retailprice) as s from part group by p_mfgr, p_brand; select p_mfgr, p_brand, s, sum(s) over w1 as s1 from mfgr_price_view window w1 as (distribute by p_mfgr sort by p_brand rows between 2 preceding and current row); {noformat} Fix is to add an explicit flag to PTFDesc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642861#comment-13642861 ] Hudson commented on HIVE-3952: -- Integrated in Hive-trunk-hadoop2 #175 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/175/]) HIVE-3952 merge map-job followed by map-reduce job (Vinod Kumar Vavilapalli via namit) (Revision 1475661) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1475661 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java * /hive/trunk/ql/src/test/queries/clientpositive/multiMapJoin1.q * /hive/trunk/ql/src/test/results/clientpositive/multiMapJoin1.q.out merge map-job followed by map-reduce job Key: HIVE-3952 URL: https://issues.apache.org/jira/browse/HIVE-3952 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Vinod Kumar Vavilapalli Fix For: 0.12.0 Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, HIVE-3952-20130424.txt Consider the query like: select count(*) FROM ( select idOne, idTwo, value FROM bigTable JOIN smallTableOne on (bigTable.idOne = smallTableOne.idOne) ) firstjoin JOIN smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and hive.auto.convert.join.noconditionaltask is set to true. The joins are collapsed into mapjoins, and it leads to a map-only job (for the map-joins) followed by a map-reduce job (for the group by). Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642862#comment-13642862 ] Hudson commented on HIVE-4409: -- Integrated in Hive-trunk-hadoop2 #175 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/175/]) HIVE-4409 Prevent incompatible column type changes (Dilip Joseph via namit) (Revision 1475783) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1475783 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig * /hive/trunk/ql/src/test/queries/clientnegative/disallow_incompatible_type_change_on1.q * /hive/trunk/ql/src/test/queries/clientnegative/disallow_incompatible_type_change_on2.q * /hive/trunk/ql/src/test/queries/clientpositive/disallow_incompatible_type_change_off.q * /hive/trunk/ql/src/test/results/clientnegative/disallow_incompatible_type_change_on1.q.out * /hive/trunk/ql/src/test/results/clientnegative/disallow_incompatible_type_change_on2.q.out * /hive/trunk/ql/src/test/results/clientpositive/disallow_incompatible_type_change_off.q.out Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Fix For: 0.12.0 Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, HIVE-4409.D10539.2.patch If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642863#comment-13642863 ] Hudson commented on HIVE-4095: -- Integrated in Hive-trunk-hadoop2 #175 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/175/]) HIVE-4095 Add exchange partition in Hive (Dheeraj Kumar Singh via namit) (Revision 1475722) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1475722 Files : * /hive/trunk/.gitignore * /hive/trunk/metastore/if/hive_metastore.thrift * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h * /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py * /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableExchangePartition.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_incomplete_partition.q * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists.q * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists2.q * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists3.q * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_partition_missing.q * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_table_missing.q * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_table_missing2.q * /hive/trunk/ql/src/test/queries/clientnegative/exchange_partition_neg_test.q * /hive/trunk/ql/src/test/queries/clientpositive/exchange_partition.q * /hive/trunk/ql/src/test/queries/clientpositive/exchange_partition2.q * /hive/trunk/ql/src/test/queries/clientpositive/exchange_partition3.q * /hive/trunk/ql/src/test/results/clientnegative/alter_view_as_select_with_partition.q.out * /hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out * /hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_incomplete_partition.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists2.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists3.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_partition_missing.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_table_missing.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_table_missing2.q.out * /hive/trunk/ql/src/test/results/clientnegative/exchange_partition_neg_test.q.out * /hive/trunk/ql/src/test/results/clientpositive/exchange_partition.q.out * /hive/trunk/ql/src/test/results/clientpositive/exchange_partition2.q.out * /hive/trunk/ql/src/test/results/clientpositive/exchange_partition3.q.out Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar
Hive-trunk-hadoop2 - Build # 175 - Still Failing
Changes for Build #138 [namit] HIVE-4289 HCatalog build fails when behind a firewall (Samuel Yuan via namit) [namit] HIVE-4281 add hive.map.groupby.sorted.testmode (Namit via Gang Tim Liu) [hashutosh] Moving hcatalog site outside of trunk [hashutosh] Moving hcatalog branches outside of trunk [hashutosh] HIVE-4259 : SEL operator created with missing columnExprMap for unions (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4156 : need to add protobuf classes to hive-exec.jar (Owen Omalley via Ashutosh Chauhan) [hashutosh] HIVE-3464 : Merging join tree may reorder joins which could be invalid (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4138 : ORC's union object inspector returns a type name that isn't parseable by TypeInfoUtils (Owen Omalley via Ashutosh Chauhan) [cws] HIVE-4119. ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails with NPE if the table is empty (Shreepadma Venugopalan via cws) [hashutosh] HIVE-4252 : hiveserver2 string representation of complex types are inconsistent with cli (Thejas Nair via Ashutosh Chauhan) [hashutosh] HIVE-4179 : NonBlockingOpDeDup does not merge SEL operators correctly (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4269 : fix handling of binary type in hiveserver2, jdbc driver (Thejas Nair via Ashutosh Chauhan) [namit] HIVE-4174 Round UDF converts BigInts to double (Chen Chun via namit) [namit] HIVE-4240 optimize hive.enforce.bucketing and hive.enforce sorting insert (Gang Tim Liu via namit) [navis] HIVE-4288 Add IntelliJ project files files to .gitignore (Roshan Naik via Navis) [namit] HIVE-4272 partition wise metadata does not work for text files [hashutosh] HIVE-896 : Add LEAD/LAG/FIRST/LAST analytical windowing functions to Hive. (Harish Butani via Ashutosh Chauhan) [namit] HIVE-4260 union_remove_12, union_remove_13 are failing on hadoop2 (Gunther Hagleitner via namit) [hashutosh] HIVE-3951 : Allow Decimal type columns in Regex Serde (Mark Grover via Ashutosh Chauhan) [namit] HIVE-4270 bug in hive.map.groupby.sorted in the presence of multiple input partitions (Namit via Gang Tim Liu) [hashutosh] HIVE-3850 : hour() function returns 12 hour clock value when using timestamp datatype (Anandha and Franklin via Ashutosh Chauhan) [hashutosh] HIVE-4122 : Queries fail if timestamp data not in expected format (Prasad Mujumdar via Ashutosh Chauhan) [hashutosh] HIVE-4170 : [REGRESSION] FsShell.close closes filesystem, removing temporary directories (Navis via Ashutosh Chauhan) [gates] HIVE-4264 Moved hcatalog trunk code up to hive/trunk/hcatalog [hashutosh] HIVE-4263 : Adjust build.xml package command to move all hcat jars and binaries into build (Alan Gates via Ashutosh Chauhan) [namit] HIVE-4258 Log logical plan tree for debugging (Navis via namit) [navis] HIVE-2264 Hive server is SHUTTING DOWN when invalid queries beeing executed [kevinwilfong] HIVE-4235. CREATE TABLE IF NOT EXISTS uses inefficient way to check if table exists. (Gang Tim Liu via kevinwilfong) [gangtimliu] HIVE-4157: ORC runs out of heap when writing (Kevin Wilfong vi Gang Tim Liu) [gangtimliu] HIVE-4155: Expose ORC's FileDump as a service [gangtimliu] HIVE-4159:RetryingHMSHandler doesn't retry in enough cases (Kevin Wilfong vi Gang Tim Liu) [namit] HIVE-4149 wrong results big outer joins with array of ints (Navis via namit) [namit] HIVE-3958 support partial scan for analyze command - RCFile (Gang Tim Liu via namit) [gates] Removing old branches to limit size of Hive downloads. [gates] Removing tags directory as we no longer need them and they're in the history. [gates] Moving HCatalog into Hive. [gates] Test that perms work for hcatalog [hashutosh] HIVE-4007 : Create abstract classes for serializer and deserializer (Namit Jain via Ashutosh Chauhan) [hashutosh] HIVE-3381 : Result of outer join is not valid (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3980 : Cleanup after 3403 (Namit Jain via Ashutosh Chauhan) [hashutosh] HIVE-4042 : ignore mapjoin hint (Namit Jain via Ashutosh Chauhan) [namit] HIVE-3348 semi-colon in comments in .q file does not work (Nick Collins via namit) [namit] HIVE-4212 sort merge join should work for outer joins for more than 8 inputs (Namit via Gang Tim Liu) [namit] HIVE-4219 explain dependency does not capture the input table (Namit via Gang Tim Liu) [kevinwilfong] HIVE-4092. Store complete names of tables in column access analyzer (Samuel Yuan via kevinwilfong) [namit] HIVE-4208 Clientpositive test parenthesis_star_by is non-deteministic (Mark Grover via namit) [cws] HIVE-4217. Fix show_create_table_*.q test failures (Carl Steinbach via cws) [namit] HIVE-4206 Sort merge join does not work for outer joins for 7 inputs (Namit via Gang Tim Liu) [kevinwilfong] HIVE-4188. TestJdbcDriver2.testDescribeTable failing consistently. (Prasad Mujumdar via kevinwilfong) [hashutosh] HIVE-3820 Consider creating a literal like D or BD for representing Decimal type constants (Gunther Hagleitner
Re: Review Request: Improve RCFile::sync(long) by 10x
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10795/#review19770 --- Ship it! Ship It! - Ashutosh Chauhan On April 26, 2013, 11:25 a.m., Gopal V wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10795/ --- (Updated April 26, 2013, 11:25 a.m.) Review request for hive, Ashutosh Chauhan and Gunther Hagleitner. Description --- Speed up RCFile::sync() by reading large blocks of data from HDFS rather than using readByte() on the input stream. This improves the loop behaviour and reduces the number of calls on the synchronized read() methods within HDFS, resulting in a 10x performance boost to this function. In real time, it converts a call that takes upto a second and brings it below 100ms, by reading 512 byte chunks instead of reading data 1 byte at a time. This addresses bug HIVE-4423. https://issues.apache.org/jira/browse/HIVE-4423 Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java d3d98d0 Diff: https://reviews.apache.org/r/10795/diff/ Testing --- ant test -Dtestcase=TestRCFile -Dmodule=ql ant test -Dtestcase=TestCliDriver -Dqfile_regex=.*rcfile.* -Dmodule=ql And benchmarking with count(1) on the store_sales rcfile table at scale=10 before: 43.8, after: 39.5 Thanks, Gopal V
[jira] [Commented] (HIVE-4423) Improve RCFile::sync(long) 10x
[ https://issues.apache.org/jira/browse/HIVE-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642944#comment-13642944 ] Ashutosh Chauhan commented on HIVE-4423: +1 will commit if tests pass Improve RCFile::sync(long) 10x -- Key: HIVE-4423 URL: https://issues.apache.org/jira/browse/HIVE-4423 Project: Hive Issue Type: Improvement Environment: Ubuntu LXC (1 SSD, 1 disk, 32 gigs of RAM) Reporter: Gopal V Assignee: Gopal V Priority: Minor Fix For: 0.11.0 Attachments: HIVE-4423.patch RCFile::sync(long) takes approx ~1 second everytime it gets called because of the inner loops in the function. From what was observed with HDFS-4710, single byte reads are an order of magnitude slower than larger 512 byte buffer reads. Even when disk I/O is buffered to this size, there is overhead due to the synchronized read() methods in BlockReaderLocal RemoteBlockReader classes. Removing the readByte() calls in RCFile.sync(long) with a readFully(512 byte) call will speed this function 10x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4404) Vectorized filter and select operators
[ https://issues.apache.org/jira/browse/HIVE-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642947#comment-13642947 ] Jitendra Nath Pandey commented on HIVE-4404: Review board entry for the patch: https://reviews.apache.org/r/10788/ Vectorized filter and select operators -- Key: HIVE-4404 URL: https://issues.apache.org/jira/browse/HIVE-4404 Project: Hive Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-4404.1.patch We need to implement filter and select operators that work on vectorized input and use vectorized expressions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3708) Add mapreduce workflow information to job configuration
[ https://issues.apache.org/jira/browse/HIVE-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-3708. Resolution: Fixed Fix Version/s: 0.12.0 Committed to trunk. Thanks, Billie! Add mapreduce workflow information to job configuration --- Key: HIVE-3708 URL: https://issues.apache.org/jira/browse/HIVE-3708 Project: Hive Issue Type: Improvement Reporter: Billie Rinaldi Fix For: 0.12.0 Attachments: HIVE-3708_1.patch, HIVE-3708.patch, HIVE-3708.patch Adding workflow properties to the job configuration would enable logging and analysis of workflows in addition to individual MapReduce jobs. Suggested properties include a workflow ID, workflow name, adjacency list connecting nodes in the workflow, and the name of the current node in the workflow. mapreduce.workflow.id - a unique ID for the workflow, ideally prepended with the application name e.g. hive_hiveQueryId mapreduce.workflow.name - a name for the workflow, to distinguish this workflow from other workflows and to group different runs of the same workflow e.g. hive query string mapreduce.workflow.adjacency - an adjacency list for the workflow graph, encoded as mapreduce.workflow.adjacency.source node = comma-separated list of target nodes mapreduce.workflow.node.name - the name of the node corresponding to this MapReduce job in the workflow adjacency list -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function
[ https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642949#comment-13642949 ] Harish Butani commented on HIVE-4019: - +1 Ability to create and drop temporary partition function --- Key: HIVE-4019 URL: https://issues.apache.org/jira/browse/HIVE-4019 Project: Hive Issue Type: New Feature Components: PTF-Windowing Reporter: Ashutosh Chauhan Assignee: Brock Noland Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, hive-4019.q Just like udf/udaf/udtf functions, user should be able to add and drop custom partitioning functions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3708) Add mapreduce workflow information to job configuration
[ https://issues.apache.org/jira/browse/HIVE-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-3708: --- Assignee: Billie Rinaldi Add mapreduce workflow information to job configuration --- Key: HIVE-3708 URL: https://issues.apache.org/jira/browse/HIVE-3708 Project: Hive Issue Type: Improvement Reporter: Billie Rinaldi Assignee: Billie Rinaldi Fix For: 0.12.0 Attachments: HIVE-3708_1.patch, HIVE-3708.patch, HIVE-3708.patch Adding workflow properties to the job configuration would enable logging and analysis of workflows in addition to individual MapReduce jobs. Suggested properties include a workflow ID, workflow name, adjacency list connecting nodes in the workflow, and the name of the current node in the workflow. mapreduce.workflow.id - a unique ID for the workflow, ideally prepended with the application name e.g. hive_hiveQueryId mapreduce.workflow.name - a name for the workflow, to distinguish this workflow from other workflows and to group different runs of the same workflow e.g. hive query string mapreduce.workflow.adjacency - an adjacency list for the workflow graph, encoded as mapreduce.workflow.adjacency.source node = comma-separated list of target nodes mapreduce.workflow.node.name - the name of the node corresponding to this MapReduce job in the workflow adjacency list -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4350) support AS keyword for table alias
[ https://issues.apache.org/jira/browse/HIVE-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4350: --- Status: Open (was: Patch Available) I got 15 failures in TestNegativeCliDriver support AS keyword for table alias -- Key: HIVE-4350 URL: https://issues.apache.org/jira/browse/HIVE-4350 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.10.0, 0.11.0 Reporter: Thejas M Nair Assignee: Matthew Weaver Attachments: HIVE-4350.D10503.1.patch, HIVE-4350.D10503.2.patch, HIVE-4350.D10503.3.patch SQL standard supports AS optional keyword, while creating an table alias. http://savage.net.au/SQL/sql-92.bnf.html#table reference Hive gives a error when the optional keyword is used - select * from tiny as t1; org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: ParseException line 1:19 mismatched input 'as' expecting EOF near 'tiny' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function
[ https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642965#comment-13642965 ] Harish Butani commented on HIVE-4019: - [~le...@hortonworks.com] thanks for starting the documentation. Not sure, maybe this works: {noformat} *** [Windowing and Analytics Functions|LanguageManual Windowingamp;Analytics] {noformat} Ability to create and drop temporary partition function --- Key: HIVE-4019 URL: https://issues.apache.org/jira/browse/HIVE-4019 Project: Hive Issue Type: New Feature Components: PTF-Windowing Reporter: Ashutosh Chauhan Assignee: Brock Noland Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, hive-4019.q Just like udf/udaf/udtf functions, user should be able to add and drop custom partitioning functions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2817) Drop any table even without privilege
[ https://issues.apache.org/jira/browse/HIVE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2817: -- Attachment: HIVE-2817.D10563.1.patch chenchun requested code review of HIVE-2817 [jira] Drop any table even without privilege. Reviewers: JIRA HIVE-2817 You can drop any table if you use fully qualified name 'database.table' even you don't have any previlige. hive set hive.security.authorization.enabled=true; hive revoke all on default from user test_user; hive drop table abc; hive drop table abc; Authorization failed:No privilege 'Drop' found for outputs { database:default, table:abc}. Use show grant to get more details. hive drop table default.abc; OK Time taken: 0.13 seconds The table and the file in /usr/hive/warehouse or external file will be deleted. If you don't have hadoop access permission on /usr/hive/warehouse or external files, you will see a hadoop access error 12/02/23 15:35:35 ERROR hive.log: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=test_user, access=WRITE, inode=/user/myetl:myetl:etl:drwxr-xr-x at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D10563 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/test/queries/clientnegative/authorization_fail_8.q ql/src/test/results/clientnegative/authorization_fail_8.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/25287/ To: JIRA, chenchun Drop any table even without privilege - Key: HIVE-2817 URL: https://issues.apache.org/jira/browse/HIVE-2817 Project: Hive Issue Type: Bug Affects Versions: 0.7.1 Reporter: Benyi Wang Assignee: Chen Chun Attachments: HIVE-2817.D10371.1.patch, HIVE-2817.D10563.1.patch You can drop any table if you use fully qualified name 'database.table' even you don't have any previlige. {code} hive set hive.security.authorization.enabled=true; hive revoke all on default from user test_user; hive drop table abc; hive drop table abc; Authorization failed:No privilege 'Drop' found for outputs { database:default, table:abc}. Use show grant to get more details. hive drop table default.abc; OK Time taken: 0.13 seconds {code} The table and the file in {{/usr/hive/warehouse}} or external file will be deleted. If you don't have hadoop access permission on {{/usr/hive/warehouse}} or external files, you will see a hadoop access error {code} 12/02/23 15:35:35 ERROR hive.log: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=test_user, access=WRITE, inode=/user/myetl:myetl:etl:drwxr-xr-x at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-4350) support AS keyword for table alias
[ https://issues.apache.org/jira/browse/HIVE-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-4350 started by Matthew Weaver. support AS keyword for table alias -- Key: HIVE-4350 URL: https://issues.apache.org/jira/browse/HIVE-4350 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.10.0, 0.11.0 Reporter: Thejas M Nair Assignee: Matthew Weaver Attachments: HIVE-4350.D10503.1.patch, HIVE-4350.D10503.2.patch, HIVE-4350.D10503.3.patch SQL standard supports AS optional keyword, while creating an table alias. http://savage.net.au/SQL/sql-92.bnf.html#table reference Hive gives a error when the optional keyword is used - select * from tiny as t1; org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: ParseException line 1:19 mismatched input 'as' expecting EOF near 'tiny' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2817) Drop any table even without privilege
[ https://issues.apache.org/jira/browse/HIVE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Chun updated HIVE-2817: Fix Version/s: 0.11.0 Affects Version/s: 0.8.0 0.9.0 0.10.0 Release Note: Fix bug drop any table even without privilege Hadoop Flags: Incompatible change Status: Patch Available (was: Open) Drop any table even without privilege - Key: HIVE-2817 URL: https://issues.apache.org/jira/browse/HIVE-2817 Project: Hive Issue Type: Bug Affects Versions: 0.10.0, 0.9.0, 0.8.0, 0.7.1 Reporter: Benyi Wang Assignee: Chen Chun Fix For: 0.11.0 Attachments: HIVE-2817.D10371.1.patch, HIVE-2817.D10563.1.patch You can drop any table if you use fully qualified name 'database.table' even you don't have any previlige. {code} hive set hive.security.authorization.enabled=true; hive revoke all on default from user test_user; hive drop table abc; hive drop table abc; Authorization failed:No privilege 'Drop' found for outputs { database:default, table:abc}. Use show grant to get more details. hive drop table default.abc; OK Time taken: 0.13 seconds {code} The table and the file in {{/usr/hive/warehouse}} or external file will be deleted. If you don't have hadoop access permission on {{/usr/hive/warehouse}} or external files, you will see a hadoop access error {code} 12/02/23 15:35:35 ERROR hive.log: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=test_user, access=WRITE, inode=/user/myetl:myetl:etl:drwxr-xr-x at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4266) Refactor HCatalog code to org.apache.hive.hcatalog
[ https://issues.apache.org/jira/browse/HIVE-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642988#comment-13642988 ] Ashutosh Chauhan commented on HIVE-4266: [~cwsteinbach] Given Alan's explanation, will you still block 0.11 for this ? Refactor HCatalog code to org.apache.hive.hcatalog -- Key: HIVE-4266 URL: https://issues.apache.org/jira/browse/HIVE-4266 Project: Hive Issue Type: Sub-task Components: HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Alan Gates Priority: Blocker Fix For: 0.11.0 Currently HCatalog code is in packages org.apache.hcatalog. It needs to now move to org.apache.hive.hcatalog. Shell classes/interface need to be created for public facing classes so that user's code does not break. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4350) support AS keyword for table alias
[ https://issues.apache.org/jira/browse/HIVE-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643031#comment-13643031 ] Matthew Weaver commented on HIVE-4350: -- Can you give more detail. All TestNegativeCliDriver queries pass in my runs. support AS keyword for table alias -- Key: HIVE-4350 URL: https://issues.apache.org/jira/browse/HIVE-4350 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.10.0, 0.11.0 Reporter: Thejas M Nair Assignee: Matthew Weaver Attachments: HIVE-4350.D10503.1.patch, HIVE-4350.D10503.2.patch, HIVE-4350.D10503.3.patch SQL standard supports AS optional keyword, while creating an table alias. http://savage.net.au/SQL/sql-92.bnf.html#table reference Hive gives a error when the optional keyword is used - select * from tiny as t1; org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: ParseException line 1:19 mismatched input 'as' expecting EOF near 'tiny' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work stopped] (HIVE-4350) support AS keyword for table alias
[ https://issues.apache.org/jira/browse/HIVE-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-4350 stopped by Matthew Weaver. support AS keyword for table alias -- Key: HIVE-4350 URL: https://issues.apache.org/jira/browse/HIVE-4350 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.10.0, 0.11.0 Reporter: Thejas M Nair Assignee: Matthew Weaver Attachments: HIVE-4350.D10503.1.patch, HIVE-4350.D10503.2.patch, HIVE-4350.D10503.3.patch SQL standard supports AS optional keyword, while creating an table alias. http://savage.net.au/SQL/sql-92.bnf.html#table reference Hive gives a error when the optional keyword is used - select * from tiny as t1; org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: ParseException line 1:19 mismatched input 'as' expecting EOF near 'tiny' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4425) HiveSessionImpl contains hard-coded version number
Ashutosh Chauhan created HIVE-4425: -- Summary: HiveSessionImpl contains hard-coded version number Key: HIVE-4425 URL: https://issues.apache.org/jira/browse/HIVE-4425 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Ashutosh Chauhan As a result doing getInfo() call on HiveServer2 currently returns current hard coded value which is 0.10.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4425) HiveSessionImpl contains hard-coded version number
[ https://issues.apache.org/jira/browse/HIVE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643046#comment-13643046 ] Ashutosh Chauhan commented on HIVE-4425: Fix is to use version info from compile time generated file common/src/gen/org/apache/hive/common/package-info.java HiveSessionImpl contains hard-coded version number -- Key: HIVE-4425 URL: https://issues.apache.org/jira/browse/HIVE-4425 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Ashutosh Chauhan As a result doing getInfo() call on HiveServer2 currently returns current hard coded value which is 0.10.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4425) HiveSessionImpl contains hard-coded version number
[ https://issues.apache.org/jira/browse/HIVE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643048#comment-13643048 ] Ashutosh Chauhan commented on HIVE-4425: Other places where it is hard-coded is: HiveDatabaseMetaData::getDatabaseProductVersion() HiveSessionImpl contains hard-coded version number -- Key: HIVE-4425 URL: https://issues.apache.org/jira/browse/HIVE-4425 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Ashutosh Chauhan As a result doing getInfo() call on HiveServer2 currently returns current hard coded value which is 0.10.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4412) PTFDesc tries serialize transient fields like OIs, etc.
[ https://issues.apache.org/jira/browse/HIVE-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4412: --- Fix Version/s: (was: 0.12.0) 0.11.0 PTFDesc tries serialize transient fields like OIs, etc. --- Key: HIVE-4412 URL: https://issues.apache.org/jira/browse/HIVE-4412 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.11.0 Attachments: HIVE-4412.D10515.1.patch windowing_expression.q shows warning messages like this. {noformat} java.lang.InstantiationException: org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableConstantDoubleObjectInspector Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... java.lang.InstantiationException: org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... java.lang.IllegalAccessException: Class sun.reflect.misc.Trampoline can not access a member of class org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableDoubleObjectInspector with modifiers Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... {noformat} hive is serializing temporary OIs in it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4426) Support statistics collection for partitioning key
Shreepadma Venugopalan created HIVE-4426: Summary: Support statistics collection for partitioning key Key: HIVE-4426 URL: https://issues.apache.org/jira/browse/HIVE-4426 Project: Hive Issue Type: Bug Reporter: Shreepadma Venugopalan Assignee: Shreepadma Venugopalan We should support the ability to collect statistics on the partitioning key column. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4427) SemanticAnalyzer throws OFFLINE_TABLE_OR_PARTITION exceptions with null context.
Morgan Phillips created HIVE-4427: - Summary: SemanticAnalyzer throws OFFLINE_TABLE_OR_PARTITION exceptions with null context. Key: HIVE-4427 URL: https://issues.apache.org/jira/browse/HIVE-4427 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Morgan Phillips Priority: Minor In SemanticAnalyzer offline tables throw a SemanticException before validate() is called. This is unnecessary, since all appropriate checks are run within the validate method, and is a bug because it throws exceptions with a null context (preventing custom offline error messages via hooks). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4425) HiveSessionImpl contains hard-coded version number
[ https://issues.apache.org/jira/browse/HIVE-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair resolved HIVE-4425. - Resolution: Duplicate Assignee: Thejas M Nair HiveSessionImpl contains hard-coded version number -- Key: HIVE-4425 URL: https://issues.apache.org/jira/browse/HIVE-4425 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Ashutosh Chauhan Assignee: Thejas M Nair As a result doing getInfo() call on HiveServer2 currently returns current hard coded value which is 0.10.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4423) Improve RCFile::sync(long) 10x
[ https://issues.apache.org/jira/browse/HIVE-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-4423: -- Labels: optimization (was: ) Release Note: Speed up RCFile::sync() by searching with a larger buffer window Status: Patch Available (was: Open) Improve RCFile::sync(long) 10x -- Key: HIVE-4423 URL: https://issues.apache.org/jira/browse/HIVE-4423 Project: Hive Issue Type: Improvement Environment: Ubuntu LXC (1 SSD, 1 disk, 32 gigs of RAM) Reporter: Gopal V Assignee: Gopal V Priority: Minor Labels: optimization Fix For: 0.11.0 Attachments: HIVE-4423.patch RCFile::sync(long) takes approx ~1 second everytime it gets called because of the inner loops in the function. From what was observed with HDFS-4710, single byte reads are an order of magnitude slower than larger 512 byte buffer reads. Even when disk I/O is buffered to this size, there is overhead due to the synchronized read() methods in BlockReaderLocal RemoteBlockReader classes. Removing the readByte() calls in RCFile.sync(long) with a readFully(512 byte) call will speed this function 10x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4160) Vectorized Query Execution in Hive
[ https://issues.apache.org/jira/browse/HIVE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-4160: -- Attachment: Hive-Vectorized-Query-Execution-Design-rev4.docx Vectorized Query Execution in Hive -- Key: HIVE-4160 URL: https://issues.apache.org/jira/browse/HIVE-4160 Project: Hive Issue Type: New Feature Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: Hive-Vectorized-Query-Execution-Design.docx, Hive-Vectorized-Query-Execution-Design-rev2.docx, Hive-Vectorized-Query-Execution-Design-rev3.docx, Hive-Vectorized-Query-Execution-Design-rev3.docx, Hive-Vectorized-Query-Execution-Design-rev3.pdf, Hive-Vectorized-Query-Execution-Design-rev4.docx, Hive-Vectorized-Query-Execution-Design-rev4.pdf The Hive query execution engine currently processes one row at a time. A single row of data goes through all the operators before the next row can be processed. This mode of processing is very inefficient in terms of CPU usage. Research has demonstrated that this yields very low instructions per cycle [MonetDB X100]. Also currently Hive heavily relies on lazy deserialization and data columns go through a layer of object inspectors that identify column type, deserialize data and determine appropriate expression routines in the inner loop. These layers of virtual method calls further slow down the processing. This work will add support for vectorized query execution to Hive, where, instead of individual rows, batches of about a thousand rows at a time are processed. Each column in the batch is represented as a vector of a primitive data type. The inner loop of execution scans these vectors very fast, avoiding method calls, deserialization, unnecessary if-then-else, etc. This substantially reduces CPU time used, and gives excellent instructions per cycle (i.e. improved processor pipeline utilization). See the attached design specification for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4160) Vectorized Query Execution in Hive
[ https://issues.apache.org/jira/browse/HIVE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-4160: -- Attachment: Hive-Vectorized-Query-Execution-Design-rev4.pdf Vectorized Query Execution in Hive -- Key: HIVE-4160 URL: https://issues.apache.org/jira/browse/HIVE-4160 Project: Hive Issue Type: New Feature Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: Hive-Vectorized-Query-Execution-Design.docx, Hive-Vectorized-Query-Execution-Design-rev2.docx, Hive-Vectorized-Query-Execution-Design-rev3.docx, Hive-Vectorized-Query-Execution-Design-rev3.docx, Hive-Vectorized-Query-Execution-Design-rev3.pdf, Hive-Vectorized-Query-Execution-Design-rev4.docx, Hive-Vectorized-Query-Execution-Design-rev4.pdf The Hive query execution engine currently processes one row at a time. A single row of data goes through all the operators before the next row can be processed. This mode of processing is very inefficient in terms of CPU usage. Research has demonstrated that this yields very low instructions per cycle [MonetDB X100]. Also currently Hive heavily relies on lazy deserialization and data columns go through a layer of object inspectors that identify column type, deserialize data and determine appropriate expression routines in the inner loop. These layers of virtual method calls further slow down the processing. This work will add support for vectorized query execution to Hive, where, instead of individual rows, batches of about a thousand rows at a time are processed. Each column in the batch is represented as a vector of a primitive data type. The inner loop of execution scans these vectors very fast, avoiding method calls, deserialization, unnecessary if-then-else, etc. This substantially reduces CPU time used, and gives excellent instructions per cycle (i.e. improved processor pipeline utilization). See the attached design specification for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4404) Vectorized filter and select operators
[ https://issues.apache.org/jira/browse/HIVE-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-4404. Resolution: Fixed Fix Version/s: vectorization-branch Committed to branch. Thanks, Jitendra! Vectorized filter and select operators -- Key: HIVE-4404 URL: https://issues.apache.org/jira/browse/HIVE-4404 Project: Hive Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: vectorization-branch Attachments: HIVE-4404.1.patch We need to implement filter and select operators that work on vectorized input and use vectorized expressions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4118) ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails when using fully qualified table name
[ https://issues.apache.org/jira/browse/HIVE-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-4118. Resolution: Duplicate Fix Version/s: 0.11.0 Duplicate of HIVE-4119 ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails when using fully qualified table name Key: HIVE-4118 URL: https://issues.apache.org/jira/browse/HIVE-4118 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.10.0 Reporter: Lenni Kuff Assignee: Shreepadma Venugopalan Fix For: 0.11.0 Computing column stats fails when using fully qualified table name. Issuing a USE db and using only the table name succeeds. {code} hive -e ANALYZE TABLE somedb.some_table COMPUTE STATISTICS FOR COLUMNS int_col org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Table somedb.some_table for which stats is gathered doesn't exist.) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2201) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:325) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:336) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111) at $Proxy9.updateTableColumnStatistics(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.update_table_column_statistics(HiveMetaStore.java:3171) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105) at $Proxy10.update_table_column_statistics(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.updateTableColumnStatistics(HiveMetaStoreClient.java:973) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74) at $Proxy11.updateTableColumnStatistics(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2198) ... 18 more {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-4118) ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails when using fully qualified table name
[ https://issues.apache.org/jira/browse/HIVE-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reopened HIVE-4118: Sorry, its not a dupe. Reopening. ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails when using fully qualified table name Key: HIVE-4118 URL: https://issues.apache.org/jira/browse/HIVE-4118 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.10.0 Reporter: Lenni Kuff Assignee: Shreepadma Venugopalan Fix For: 0.11.0 Computing column stats fails when using fully qualified table name. Issuing a USE db and using only the table name succeeds. {code} hive -e ANALYZE TABLE somedb.some_table COMPUTE STATISTICS FOR COLUMNS int_col org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Table somedb.some_table for which stats is gathered doesn't exist.) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2201) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:325) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:336) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111) at $Proxy9.updateTableColumnStatistics(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.update_table_column_statistics(HiveMetaStore.java:3171) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105) at $Proxy10.update_table_column_statistics(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.updateTableColumnStatistics(HiveMetaStoreClient.java:973) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74) at $Proxy11.updateTableColumnStatistics(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2198) ... 18 more {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4118) ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails when using fully qualified table name
[ https://issues.apache.org/jira/browse/HIVE-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4118: --- Fix Version/s: (was: 0.11.0) ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails when using fully qualified table name Key: HIVE-4118 URL: https://issues.apache.org/jira/browse/HIVE-4118 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.10.0 Reporter: Lenni Kuff Assignee: Shreepadma Venugopalan Computing column stats fails when using fully qualified table name. Issuing a USE db and using only the table name succeeds. {code} hive -e ANALYZE TABLE somedb.some_table COMPUTE STATISTICS FOR COLUMNS int_col org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Table somedb.some_table for which stats is gathered doesn't exist.) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2201) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:325) at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:336) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111) at $Proxy9.updateTableColumnStatistics(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.update_table_column_statistics(HiveMetaStore.java:3171) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105) at $Proxy10.update_table_column_statistics(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.updateTableColumnStatistics(HiveMetaStoreClient.java:973) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74) at $Proxy11.updateTableColumnStatistics(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2198) ... 18 more {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-3682: -- Attachment: HIVE-3682.D10275.2.patch khorgath updated the revision HIVE-3682 [jira] when output hive table to file,users should could have a separator of their own choice. Updated to reflect a couple of review comments: - Reused CreateTableDesc instead of creating LocalDirectoryDesc - Removed LocalDirectoryDesc - Still needs a separate function to set parameters inside the CreateTableDesc though, because of NPEs in expectations of fields like InputFormat inside CreateTableDesc. I can loosen those checks but not without worrying about whether something else will break because of that(and it does, with some minimal testing.) - Have updated tests to do things like projections, which causes an MR job - Have not updated to reflect HIVE-4369, because I can't get that without merging with trunk, and that means I can't upload using arc to reviewboard. I will update the main jira with a svn patch with that changed. Reviewers: ashutoshc, JIRA, omalley REVISION DETAIL https://reviews.facebook.net/D10275 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D10275?vs=32139id=33039#toc AFFECTED FILES data/files/array_table.txt data/files/map_table.txt ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q ql/src/test/results/clientpositive/insert_overwrite_local_directory_1.q.out To: JIRA, ashutoshc, omalley, khorgath when output hive table to file,users should could have a separator of their own choice -- Key: HIVE-3682 URL: https://issues.apache.org/jira/browse/HIVE-3682 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.8.1 Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux java version 1.6.0_25 hadoop-0.20.2-cdh3u0 hive-0.8.1 Reporter: caofangkun Assignee: Sushanth Sowmyan Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, HIVE-3682.D10275.2.patch, HIVE-3682.with.serde.patch By default,when output hive table to file ,columns of the Hive table are separated by ^A character (that is \001). But indeed users should have the right to set a seperator of their own choice. Usage Example: create table for_test (key string, value string); load data local inpath './in1.txt' into table for_test select * from for_test; UT-01:default separator is \001 line separator is \n insert overwrite local directory './test-01' select * from src ; create table array_table (a arraystring, b arraystring) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ','; load data local inpath ../hive/examples/files/arraytest.txt overwrite into table table2; CREATE TABLE map_table (foo STRING , bar MAPSTRING, STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':' STORED AS TEXTFILE; UT-02:defined field separator as ':' insert overwrite local directory './test-02' row format delimited FIELDS TERMINATED BY ':' select * from src ; UT-03: line separator DO NOT ALLOWED to define as other separator insert overwrite local directory './test-03' row format delimited FIELDS TERMINATED BY ':' select * from src ; UT-04: define map separators insert overwrite local directory './test-04' row format delimited FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':' select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-3682: -- Attachment: HIVE-3682.D10275.3.patch khorgath updated the revision HIVE-3682 [jira] when output hive table to file,users should could have a separator of their own choice. Updated to reflect HIVE-4369 change as well Reviewers: ashutoshc, JIRA, omalley REVISION DETAIL https://reviews.facebook.net/D10275 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D10275?vs=33039id=33045#toc AFFECTED FILES data/files/array_table.txt data/files/map_table.txt ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q ql/src/test/results/clientpositive/insert_overwrite_local_directory_1.q.out To: JIRA, ashutoshc, omalley, khorgath when output hive table to file,users should could have a separator of their own choice -- Key: HIVE-3682 URL: https://issues.apache.org/jira/browse/HIVE-3682 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.8.1 Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux java version 1.6.0_25 hadoop-0.20.2-cdh3u0 hive-0.8.1 Reporter: caofangkun Assignee: Sushanth Sowmyan Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.with.serde.patch By default,when output hive table to file ,columns of the Hive table are separated by ^A character (that is \001). But indeed users should have the right to set a seperator of their own choice. Usage Example: create table for_test (key string, value string); load data local inpath './in1.txt' into table for_test select * from for_test; UT-01:default separator is \001 line separator is \n insert overwrite local directory './test-01' select * from src ; create table array_table (a arraystring, b arraystring) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ','; load data local inpath ../hive/examples/files/arraytest.txt overwrite into table table2; CREATE TABLE map_table (foo STRING , bar MAPSTRING, STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':' STORED AS TEXTFILE; UT-02:defined field separator as ':' insert overwrite local directory './test-02' row format delimited FIELDS TERMINATED BY ':' select * from src ; UT-03: line separator DO NOT ALLOWED to define as other separator insert overwrite local directory './test-03' row format delimited FIELDS TERMINATED BY ':' select * from src ; UT-04: define map separators insert overwrite local directory './test-04' row format delimited FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':' select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-3682: --- Fix Version/s: 0.11.0 Status: Patch Available (was: Open) when output hive table to file,users should could have a separator of their own choice -- Key: HIVE-3682 URL: https://issues.apache.org/jira/browse/HIVE-3682 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.8.1 Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux java version 1.6.0_25 hadoop-0.20.2-cdh3u0 hive-0.8.1 Reporter: caofangkun Assignee: Sushanth Sowmyan Fix For: 0.11.0 Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.with.serde.patch By default,when output hive table to file ,columns of the Hive table are separated by ^A character (that is \001). But indeed users should have the right to set a seperator of their own choice. Usage Example: create table for_test (key string, value string); load data local inpath './in1.txt' into table for_test select * from for_test; UT-01:default separator is \001 line separator is \n insert overwrite local directory './test-01' select * from src ; create table array_table (a arraystring, b arraystring) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ','; load data local inpath ../hive/examples/files/arraytest.txt overwrite into table table2; CREATE TABLE map_table (foo STRING , bar MAPSTRING, STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':' STORED AS TEXTFILE; UT-02:defined field separator as ':' insert overwrite local directory './test-02' row format delimited FIELDS TERMINATED BY ':' select * from src ; UT-03: line separator DO NOT ALLOWED to define as other separator insert overwrite local directory './test-03' row format delimited FIELDS TERMINATED BY ':' select * from src ; UT-04: define map separators insert overwrite local directory './test-04' row format delimited FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':' select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dilip Joseph updated HIVE-4409: --- Description: If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) The ALTER TABLE statement is blocked if you set the following parameter, introduced int the fix to this JIRA: SET hive.metastore.disallow.incompatible.col.type.changes=true; was: If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Fix For:
[jira] [Commented] (HIVE-4420) HCatalog unit tests stop after a failure
[ https://issues.apache.org/jira/browse/HIVE-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643275#comment-13643275 ] Sushanth Sowmyan commented on HIVE-4420: Looks good to me, I've faked a test failure and am running tests. If tests continue to run, I will go ahead and commit. One important note to people visiting this bug later - the patch provided is for hcatalog/build.xml, not the root build.xml. HCatalog unit tests stop after a failure Key: HIVE-4420 URL: https://issues.apache.org/jira/browse/HIVE-4420 Project: Hive Issue Type: Bug Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.12.0 Attachments: HIVE-4420.patch HCatalog splits it's different components (core, hcatalog-pig-adaptor, webhcat, etc.) into separate build.xml files, all controlled by the top build.xml. A side effect of this is that when the user does ant test and the top build file is iterating through the tests, if one of the component's test targets fail then unit tests are not run for subsequent components. This means a failure in one component can mask issues in others. test should run on all the components even when there is a failure in one of the components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4428) Misspelling in describe extended output
Alan Gates created HIVE-4428: Summary: Misspelling in describe extended output Key: HIVE-4428 URL: https://issues.apache.org/jira/browse/HIVE-4428 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Alan Gates Priority: Trivial Fix For: 0.12.0 The output of describe extended includes: colelction.delim. This should be collection.delim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
How to pass config from qfile to Hive Metastore
Hi Dear all, I want to set a configuration in file and pass it to Hive Metastore for example logic in HiveAlterHandler.java. In order to do that, this configuration should be in HiveConf.metaVars. But, a simple test got NPE. Anyone has experience to pass config from qfile to Hive metastore? Attached has status.q. It has set hive.metastore.ds.retry.interval=2000 which is part of HiveConf.metaVars. Attached has error.txt. If we remove the config line from status.q, it works. Thanks Tim 2013-04-26 14:34:41,603 ERROR exec.Task (SessionState.java:printError(388)) - FAILED: Error in metadata: Unable to fetch table srcpart org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table srcpart at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:957) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:891) at org.apache.hadoop.hive.ql.exec.DDLTask.createTableLike(DDLTask.java:3803) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:279) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1355) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1139) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:945) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats60(TestCliDriver.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:154) at junit.framework.TestCase.runBare(TestCase.java:127) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) Caused by: java.lang.NullPointerException at org.datanucleus.sco.simple.Set.init(Set.java:68) at org.datanucleus.sco.backed.Set.init(Set.java:94) at org.datanucleus.sco.backed.Map.entrySet(Map.java:418) at org.apache.hadoop.hive.metastore.api.SerDeInfo.init(SerDeInfo.java:157) at org.apache.hadoop.hive.metastore.api.StorageDescriptor.init(StorageDescriptor.java:256) at org.apache.hadoop.hive.metastore.api.Table.init(Table.java:260) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.deepCopy(HiveMetaStoreClient.java:1177) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:854) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74) at $Proxy7.getTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:949) ... 30 more 2013-04-26 14:34:41,603 DEBUG exec.DDLTask (DDLTask.java:execute(459)) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table srcpart at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:957) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:891) at org.apache.hadoop.hive.ql.exec.DDLTask.createTableLike(DDLTask.java:3803) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:279) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at
[jira] [Commented] (HIVE-4420) HCatalog unit tests stop after a failure
[ https://issues.apache.org/jira/browse/HIVE-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643314#comment-13643314 ] Sushanth Sowmyan commented on HIVE-4420: Tested, works, and in fact, will also bypass and continue if there is a compile error in one group of tests. Potent! I'm waiting for the test run to finish to see if the overall test task results in a BUILD SUCCESSFUL(not expected) or a BUILD FAILURE(expected). HCatalog unit tests stop after a failure Key: HIVE-4420 URL: https://issues.apache.org/jira/browse/HIVE-4420 Project: Hive Issue Type: Bug Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.12.0 Attachments: HIVE-4420.patch HCatalog splits it's different components (core, hcatalog-pig-adaptor, webhcat, etc.) into separate build.xml files, all controlled by the top build.xml. A side effect of this is that when the user does ant test and the top build file is iterating through the tests, if one of the component's test targets fail then unit tests are not run for subsequent components. This means a failure in one component can mask issues in others. test should run on all the components even when there is a failure in one of the components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4420) HCatalog unit tests stop after a failure
[ https://issues.apache.org/jira/browse/HIVE-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643343#comment-13643343 ] Sushanth Sowmyan commented on HIVE-4420: +1, tests ran fully and ended with a BUILD FAILURE. Will commit to 0.11 and trunk. HCatalog unit tests stop after a failure Key: HIVE-4420 URL: https://issues.apache.org/jira/browse/HIVE-4420 Project: Hive Issue Type: Bug Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.12.0 Attachments: HIVE-4420.patch HCatalog splits it's different components (core, hcatalog-pig-adaptor, webhcat, etc.) into separate build.xml files, all controlled by the top build.xml. A side effect of this is that when the user does ant test and the top build file is iterating through the tests, if one of the component's test targets fail then unit tests are not run for subsequent components. This means a failure in one component can mask issues in others. test should run on all the components even when there is a failure in one of the components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4420) HCatalog unit tests stop after a failure
[ https://issues.apache.org/jira/browse/HIVE-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643345#comment-13643345 ] Sushanth Sowmyan commented on HIVE-4420: Committed. Thanks, Alan! HCatalog unit tests stop after a failure Key: HIVE-4420 URL: https://issues.apache.org/jira/browse/HIVE-4420 Project: Hive Issue Type: Bug Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.12.0 Attachments: HIVE-4420.patch HCatalog splits it's different components (core, hcatalog-pig-adaptor, webhcat, etc.) into separate build.xml files, all controlled by the top build.xml. A side effect of this is that when the user does ant test and the top build file is iterating through the tests, if one of the component's test targets fail then unit tests are not run for subsequent components. This means a failure in one component can mask issues in others. test should run on all the components even when there is a failure in one of the components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4420) HCatalog unit tests stop after a failure
[ https://issues.apache.org/jira/browse/HIVE-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-4420: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks Sushanth. HCatalog unit tests stop after a failure Key: HIVE-4420 URL: https://issues.apache.org/jira/browse/HIVE-4420 Project: Hive Issue Type: Bug Components: Build Infrastructure, HCatalog Affects Versions: 0.11.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.12.0 Attachments: HIVE-4420.patch HCatalog splits it's different components (core, hcatalog-pig-adaptor, webhcat, etc.) into separate build.xml files, all controlled by the top build.xml. A side effect of this is that when the user does ant test and the top build file is iterating through the tests, if one of the component's test targets fail then unit tests are not run for subsequent components. This means a failure in one component can mask issues in others. test should run on all the components even when there is a failure in one of the components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-4427) SemanticAnalyzer throws OFFLINE_TABLE_OR_PARTITION exceptions with null context.
[ https://issues.apache.org/jira/browse/HIVE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Morgan Phillips reassigned HIVE-4427: - Assignee: Morgan Phillips SemanticAnalyzer throws OFFLINE_TABLE_OR_PARTITION exceptions with null context. Key: HIVE-4427 URL: https://issues.apache.org/jira/browse/HIVE-4427 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Morgan Phillips Assignee: Morgan Phillips Priority: Minor Original Estimate: 24h Remaining Estimate: 24h In SemanticAnalyzer offline tables throw a SemanticException before validate() is called. This is unnecessary, since all appropriate checks are run within the validate method, and is a bug because it throws exceptions with a null context (preventing custom offline error messages via hooks). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 2079 - Still Failing
Changes for Build #2047 Changes for Build #2048 [gangtimliu] HIVE-4298: add tests for distincts for hive.map.groutp.sorted. (Namit via Gang Tim Liu) [hashutosh] HIVE-4128 : Support avg(decimal) (Brock Noland via Ashutosh Chauhan) [kevinwilfong] HIVE-4151. HiveProfiler NPE with ScriptOperator. (Pamela Vagata via kevinwilfong) Changes for Build #2049 [hashutosh] HIVE-3985 : Update new UDAFs introduced for Windowing to work with new Decimal Type (Brock Noland via Ashutosh Chauhan) [hashutosh] HIVE-3840 : hive cli null representation in output is inconsistent (Thejas Nair via Ashutosh Chauhan) [hashutosh] HIVE-4262 : fix last_value UDAF behavior (Harish Butani via Ashutosh Chauhan) [hashutosh] HIVE-4292 : hiveserver2 should support -hiveconf commandline parameter (Thejas Nair via Ashutosh Chauhan) Changes for Build #2050 [hashutosh] HIVE-3908 : create view statement's outputs contains the view and a temporary dir. (Prasad Mujumdar via Ashutosh Chauhan) [hashutosh] HIVE-4302 : Fix how RowSchema and RowResolver are set on ReduceSinkOp that precedes PTFOp (Harish Butani via Ashutosh Chauhan) Changes for Build #2051 [hashutosh] HIVE-3992 : Hive RCFile::sync(long) does a sub-sequence linear search for sync blocks (Gopal V via Ashutosh Chauhan) Changes for Build #2052 Changes for Build #2053 [navis] Missing test results from HIVE-1953 (Vikram Dixit K via Navis) [namit] HIVE-4314 Result of mapjoin_test_outer.q is not deterministic (Navis via namit) [navis] HIVE-1953 Hive should process comments in CliDriver (Vikram Dixit K via Navis) [navis] HIVE-3308 Mixing avro and snappy gives null values (Bennie Schut via Navis) [hashutosh] HIVE-4311 : DOS line endings in auto_join26.q (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-2340 : optimize orderby followed by a groupby (Navis via Ashutosh Chauhan) Changes for Build #2054 [khorgath] HCATALOG-632 Fixing ORC File usage with HCatalog Changes for Build #2055 [hashutosh] HIVE-4107 : Update Hive 0.10.0 RELEASE_NOTES.txt (Thejas Nair via Ashutosh Chauhan) [hashutosh] HIVE-4271 : Limit precision of decimal type (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4319 : Revert changes checked-in as part of 1953 (Vikram Dixit via Ashutosh Chauhan) Changes for Build #2056 [hashutosh] HIVE-4078 : Delay the serialize-deserialize pair in CommonJoinTaskDispatcher (Gopal V via Ashutosh Chauhan) [gangtimliu] HIVE-4337: Update list bucketing test results (Samuel Yuan via Gang Tim Liu) [hashutosh] HIVE-4306 : PTFDeserializer should reconstruct OIs based on InputOI passed to PTFOperator (Harish Butani and Prajakta Kalmegh via Ashutosh Chauhan) [hashutosh] HIVE-4334 : ctas test on hadoop 2 has outdated golden file (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4332 : orc*.q tests fail on hadoop 2 (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4120 : Implement decimal encoding for ORC (Owen Omalley via Ashutosh Chauhan) [hashutosh] HIVE-4303 : 2 empty java files in hcatalog (Owen Omalley via Ashutosh Chauhan) [hashutosh] HIVE-4335 : serde_regex test fails on hadoop 2 (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4325 : Merge HCat NOTICE file with Hive NOTICE file (Alan Gates via Ashutosh Chauhan) [gates] Moving CHANGES.txt from under trunk to hcatalog-historical [gates] HIVE-4236 - Clean up remaining items in hive/hcatalog/historical/trunk (gates) Changes for Build #2057 [namit] HIVE-4328 Change default bigtable selection policy for sort-merge joins (Namit Jain via Ashutosh Chauhan) [hashutosh] Preparing for 0.12 development [hashutosh] HIVE-4323 : sqlline dependency is not required (Thejas Nair via Ashutosh Chauhan) [namit] HIVE-4336 Selecting from a view, and another view that also selects from that view fails (Kevin Wilfong via namit) Changes for Build #2058 [namit] HIVE-4241 optimize hive.enforce.sorting and hive.enforce bucketing join (Namit Jain via Gang Tim Liu) Changes for Build #2059 [namit] HIVE-4294 Single sourced multi query cannot handle lateral view (Navis via namit) Changes for Build #2060 Changes for Build #2061 [hashutosh] HIVE-4339 : build fails after branch (hcatalog version not updated) (Gunther Hagleitner via Ashutosh Chauhan) Changes for Build #2062 [hashutosh] HIVE-4296 : ant thriftif fails on hcatalog (Roshan Naik via Ashutosh Chauhan) [gangtimliu] HIVE-4322: SkewedInfo in Metastore Thrift API cannot be deserialized in Python (Samuel Yuan via Gang Tim Liu) Changes for Build #2063 Changes for Build #2064 Changes for Build #2065 [navis] HIVE-3179 HBase Handler doesn't handle NULLs properly (Lars Francke via Navis) Changes for Build #2066 [cws] HIVE-4268. Beeline should support the -f option (Rob Weltman via cws) Changes for Build #2067 [hashutosh] HIVE-4275 : Hive does not differentiate scheme and authority in file uris (Vikram Dixit via Ashutosh Chauhan) [hashutosh] HIVE-4352 : Guava not getting included in build
[jira] [Commented] (HIVE-4358) Check for Map side processing in PTFOp is no longer valid
[ https://issues.apache.org/jira/browse/HIVE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643367#comment-13643367 ] Hudson commented on HIVE-4358: -- Integrated in Hive-trunk-h0.21 #2079 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2079/]) HIVE-4358 : Check for Map side processing in PTFOp is no longer valid (Harish Butani via Ashutosh Chauhan) (Revision 1475880) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1475880 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java * /hive/trunk/ql/src/test/queries/clientpositive/windowing.q * /hive/trunk/ql/src/test/results/clientpositive/windowing.q.out Check for Map side processing in PTFOp is no longer valid - Key: HIVE-4358 URL: https://issues.apache.org/jira/browse/HIVE-4358 Project: Hive Issue Type: Bug Components: PTF-Windowing Reporter: Harish Butani Assignee: Harish Butani Fix For: 0.12.0 Attachments: HIVE-4358.D10473.1.patch With the changes for ReduceSinkDedup it is no longer true that a non Map-side PTF Operator is preceded by an ExtractOp. For e.g. following query can produce the issue: {noformat} create view IF NOT EXISTS mfgr_price_view as select p_mfgr, p_brand, sum(p_retailprice) as s from part group by p_mfgr, p_brand; select p_mfgr, p_brand, s, sum(s) over w1 as s1 from mfgr_price_view window w1 as (distribute by p_mfgr sort by p_brand rows between 2 preceding and current row); {noformat} Fix is to add an explicit flag to PTFDesc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
[ https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643368#comment-13643368 ] Hudson commented on HIVE-4424: -- Integrated in Hive-trunk-h0.21 #2079 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2079/]) HIVE-4424 MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 (Namit Jain) (Revision 1476119) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476119 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 --- Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4342) NPE for query involving UNION ALL with nested JOIN and UNION ALL
[ https://issues.apache.org/jira/browse/HIVE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643407#comment-13643407 ] Mihir Kulkarni commented on HIVE-4342: -- Thanks a lot [~navis] and [~namit] for fixing this issue. NPE for query involving UNION ALL with nested JOIN and UNION ALL Key: HIVE-4342 URL: https://issues.apache.org/jira/browse/HIVE-4342 Project: Hive Issue Type: Bug Components: Logging, Metastore, Query Processor Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Assignee: Navis Priority: Critical Fix For: 0.11.0 Attachments: HIVE-4342.D10407.1.patch, HiveCommands.txt, Query.txt, sourceData1.txt, sourceData2.txt UNION ALL query with JOIN in first part and another UNION ALL in second part gives NPE. bq. JOIN UNION ALL bq. UNION ALL Attachments: 1. HiveCommands.txt : command script to setup schema for query under consideration. 2. sourceData1.txt and sourceData2.txt : required for above command script. 3. Query.txt : Exact query which produces NPE. NOTE: you will need to update path to sourceData1.txt and sourceData2.txt in the HiveCommands.txt to suit your environment. Attached files contain the schema and exact query which fails on Hive 0.9. It is worthwhile to note that the same query executes successfully on Hive 0.7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4429) Nested ORDER BY produces incorrect result
Mihir Kulkarni created HIVE-4429: Summary: Nested ORDER BY produces incorrect result Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Attachment: Test_Data.txt HiveQuery.txt Hive_Command_Script.txt 1) Hive_Command_Script.txt This file contains the commands to setup schema and load data. 2) HiveQuery.txt This file contains the queries which reproduce the bug. 3) Test_Data.txt File containing data to load into the schema required by Hive_Command_Script.txt Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Description: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=Bar.java|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. was: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=Bar.java|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Description: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=query.sql|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. was: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=Bar.java|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=query.sql|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Description: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=Query.sql|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. was: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=query.sql|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=Query.sql|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Description: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. was: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Description: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. was: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:title=Query.sql|borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Description: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. was: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} --- On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Kulkarni updated HIVE-4429: - Description: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. was: Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} --- Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
hive pull request: Intial Patch for HIVE-4331
GitHub user virajb opened a pull request: https://github.com/apache/hive/pull/8 Intial Patch for HIVE-4331 You can merge this pull request into a Git repository by running: $ git pull https://github.com/virajb/hive HIVE-4331 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/8.patch
[jira] [Created] (HIVE-4430) Semantic analysis fails in presence of certain literals in on clause
Kevin Wilfong created HIVE-4430: --- Summary: Semantic analysis fails in presence of certain literals in on clause Key: HIVE-4430 URL: https://issues.apache.org/jira/browse/HIVE-4430 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.12.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Minor When users include a bigint literal (a number suffixed with 'L') in the conditions in the on clause the query will fail with, e.g. FAILED: SemanticException 0L encountered with 0 children I haven't tried it yet, but I suspect the same is true for other, lesser used literals. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4431) Implement vectorized string concatenation
Eric Hanson created HIVE-4431: - Summary: Implement vectorized string concatenation Key: HIVE-4431 URL: https://issues.apache.org/jira/browse/HIVE-4431 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Include Col-Col, Scalar-Col, and Col-Scalar logic -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4431) Implement vectorized string concatenation
[ https://issues.apache.org/jira/browse/HIVE-4431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-4431: -- Assignee: Eric Hanson Implement vectorized string concatenation - Key: HIVE-4431 URL: https://issues.apache.org/jira/browse/HIVE-4431 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Include Col-Col, Scalar-Col, and Col-Scalar logic -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4430) Semantic analysis fails in presence of certain literals in on clause
[ https://issues.apache.org/jira/browse/HIVE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4430: -- Attachment: HIVE-4430.HIVE-4430.HIVE-4430.HIVE-4430.D10587.1.patch kevinwilfong requested code review of HIVE-4430 [jira] Semantic analysis fails in presence of certain literals in on clause. Reviewers: gangtimliu, njain https://issues.apache.org/jira/browse/HIVE-4430 When users include a bigint literal (a number suffixed with 'L') in the conditions in the on clause the query will fail with, e.g. FAILED: SemanticException 0L encountered with 0 children I haven't tried it yet, but I suspect the same is true for other, lesser used literals. TEST PLAN Added testcases for lesser used literals. REVISION DETAIL https://reviews.facebook.net/D10587 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/test/queries/clientpositive/join_literals.q ql/src/test/results/clientpositive/join_literals.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/25341/ To: gangtimliu, njain, kevinwilfong Cc: JIRA Semantic analysis fails in presence of certain literals in on clause Key: HIVE-4430 URL: https://issues.apache.org/jira/browse/HIVE-4430 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.12.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Minor Attachments: HIVE-4430.HIVE-4430.HIVE-4430.HIVE-4430.D10587.1.patch When users include a bigint literal (a number suffixed with 'L') in the conditions in the on clause the query will fail with, e.g. FAILED: SemanticException 0L encountered with 0 children I haven't tried it yet, but I suspect the same is true for other, lesser used literals. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4430) Semantic analysis fails in presence of certain literals in on clause
[ https://issues.apache.org/jira/browse/HIVE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-4430: Status: Patch Available (was: Open) Semantic analysis fails in presence of certain literals in on clause Key: HIVE-4430 URL: https://issues.apache.org/jira/browse/HIVE-4430 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.12.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Minor Attachments: HIVE-4430.HIVE-4430.HIVE-4430.HIVE-4430.D10587.1.patch When users include a bigint literal (a number suffixed with 'L') in the conditions in the on clause the query will fail with, e.g. FAILED: SemanticException 0L encountered with 0 children I haven't tried it yet, but I suspect the same is true for other, lesser used literals. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4432) Follow-up to HIVE-4322 - make metastore API changes backwards compatible
[ https://issues.apache.org/jira/browse/HIVE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samuel Yuan updated HIVE-4432: -- Description: Right now the fix for HIVE-4322 makes different versions of the metastore server and client incompatible with each other. This can make deployment very painful. Follow-up to HIVE-4322 - make metastore API changes backwards compatible Key: HIVE-4432 URL: https://issues.apache.org/jira/browse/HIVE-4432 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Reporter: Samuel Yuan Assignee: Samuel Yuan Right now the fix for HIVE-4322 makes different versions of the metastore server and client incompatible with each other. This can make deployment very painful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643453#comment-13643453 ] Lianhui Wang commented on HIVE-4429: hi, Mihir Kulkarni i run the first sql of your cases, but in my hive-0.9, it produces correct result.it is the following. 30.01.0 20.01.0 10.01.0 30.02.0 20.02.0 10.02.0 30.03.0 20.03.0 10.03.0 60.04.0 50.04.0 40.04.0 60.05.0 50.05.0 40.05.0 60.06.0 50.06.0 40.06.0 so can you tell which version you used. Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4433) Fix C++ Thrift bindings broken in HIVE-4322
Carl Steinbach created HIVE-4433: Summary: Fix C++ Thrift bindings broken in HIVE-4322 Key: HIVE-4433 URL: https://issues.apache.org/jira/browse/HIVE-4433 Project: Hive Issue Type: Bug Components: Thrift API Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4433) Fix C++ Thrift bindings broken in HIVE-4322
[ https://issues.apache.org/jira/browse/HIVE-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-4433: - Affects Version/s: 0.11.0 Fix C++ Thrift bindings broken in HIVE-4322 --- Key: HIVE-4433 URL: https://issues.apache.org/jira/browse/HIVE-4433 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.11.0 Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4433) Fix C++ Thrift bindings broken in HIVE-4322
[ https://issues.apache.org/jira/browse/HIVE-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643463#comment-13643463 ] Carl Steinbach commented on HIVE-4433: -- HIVE-4322 added a SkewedValueList structure to the Metastore Thrift IDL, along with a map field where this structure is used as a key: {noformat} // Workaround for HIVE-4322struct SkewedValueList { 1: liststring skewedValueList } // this object holds all the information about skewed table struct SkewedInfo { 1: liststring skewedColNames, // skewed column names 2: listliststring skewedColValues, //skewed values 3: mapSkewedValueList, string skewedColValueLocationMaps, //skewed value to location mappings } {noformat} It turns out that this change breaks the C++ Thrift bindings (and possibly also the bindings for other languages). According to the Thrift documentation [here|http://thrift.apache.org/docs/types/] it's a really bad idea to use a struct or container type as a map key: {quote} N.B.: For maximal compatibility, the key type for map should be a basic type rather than a struct or container type. There are some languages which do not support more complex key types in their native map types. In addition the JSON protocol only supports key types that are base types. {quote} I haven't had much time to look into this, but my hunch is that we probably need to backout HIVE-4322. Fix C++ Thrift bindings broken in HIVE-4322 --- Key: HIVE-4433 URL: https://issues.apache.org/jira/browse/HIVE-4433 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Affects Versions: 0.11.0 Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4433) Fix C++ Thrift bindings broken in HIVE-4322
[ https://issues.apache.org/jira/browse/HIVE-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-4433: - Priority: Blocker (was: Major) Fix C++ Thrift bindings broken in HIVE-4322 --- Key: HIVE-4433 URL: https://issues.apache.org/jira/browse/HIVE-4433 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Affects Versions: 0.11.0 Reporter: Carl Steinbach Priority: Blocker -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4433) Fix C++ Thrift bindings broken in HIVE-4322
[ https://issues.apache.org/jira/browse/HIVE-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643466#comment-13643466 ] Carl Steinbach commented on HIVE-4433: -- Marking this as a blocker for 0.11.0. Fix C++ Thrift bindings broken in HIVE-4322 --- Key: HIVE-4433 URL: https://issues.apache.org/jira/browse/HIVE-4433 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Affects Versions: 0.11.0 Reporter: Carl Steinbach Priority: Blocker -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4432) Follow-up to HIVE-4322 - make metastore API changes backwards compatible
[ https://issues.apache.org/jira/browse/HIVE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643467#comment-13643467 ] Carl Steinbach commented on HIVE-4432: -- It turns out that HIVE-4322 also broke the C++ Thrift bindings. I created HIVE-4433 to track this other issue. Follow-up to HIVE-4322 - make metastore API changes backwards compatible Key: HIVE-4432 URL: https://issues.apache.org/jira/browse/HIVE-4432 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Reporter: Samuel Yuan Assignee: Samuel Yuan Right now the fix for HIVE-4322 makes different versions of the metastore server and client incompatible with each other. This can make deployment very painful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4429) Nested ORDER BY produces incorrect result
[ https://issues.apache.org/jira/browse/HIVE-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643483#comment-13643483 ] Mihir Kulkarni commented on HIVE-4429: -- [~lianhuiwang] Isn't the output you mentioned above wrong? Because my first sql says *ORDER BY d0 ASC, d1 DESC* And the output you get is *d1 ASC d0 DESC* Nested ORDER BY produces incorrect result - Key: HIVE-4429 URL: https://issues.apache.org/jira/browse/HIVE-4429 Project: Hive Issue Type: Bug Components: Query Processor, SQL, UDF Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Priority: Critical Attachments: Hive_Command_Script.txt, HiveQuery.txt, Test_Data.txt Nested ORDER BY clause doesn't honor the outer one in specific case. The below query produces result which honors only the inner ORDER BY clause. (it produces only 1 MapRed job) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 ASC, d1 DESC; {code} On the other hand the query below honors the outer ORDER BY clause which produces the correct result. (it produces 2 MapRed jobs) {code:borderStyle=solid} SELECT alias.b0 as d0, alias.b1 as d1 FROM (SELECT test.a0 as b0, test.a1 as b1 FROM test ORDER BY b1 ASC, b0 DESC) alias ORDER BY d0 DESC, d1 DESC; {code} Any other combination of nested ORDER BY clauses does produce the correct result. Please see attachments for query, schema and Hive Commands for reprocase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4373) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect
[ https://issues.apache.org/jira/browse/HIVE-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-4373: Attachment: HIVE-4373.1.patch HIVE-4373.1.patch changes - HiveSessionImpl.getInfo populates the current version using HiveVersionInfo.getVersion() - Test case also checks for format of version. - saveVersion.sh produces the package-info.java in the right directory, ie directory with HiveVersionInfo.java, so that its package is annotated - changed build.xml to build package-info.java Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect Key: HIVE-4373 URL: https://issues.apache.org/jira/browse/HIVE-4373 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Deepesh Khandelwal Assignee: Thejas M Nair Priority: Minor Fix For: 0.11.0 Attachments: HIVE-4373.1.patch When running beeline {code} % beeline -u 'jdbc:hive2://localhost:1' -n hive -p passwd -d org.apache.hive.jdbc.HiveDriver Connecting to jdbc:hive2://localhost:1 Connected to: Hive (version 0.10.0) Driver: Hive (version 0.11.0) Transaction isolation: TRANSACTION_REPEATABLE_READ {code} The Hive version in the Connected to: string says 0.10.0 instead of 0.11.0. Looking at the code it seems that the version is hardcoded at two places: line 250 in jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java line 833 in jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-4373 - Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10812/ --- Review request for hive. Description --- HIVE-4373 This addresses bug HIVE-4373. https://issues.apache.org/jira/browse/HIVE-4373 Diffs - common/build.xml 731f26e common/src/scripts/saveVersion.sh a917d4a jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java 3d1a969 jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 938dd9a service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 18594cb Diff: https://reviews.apache.org/r/10812/diff/ Testing --- Unit test included. Also tested with remote HS2 server. Thanks, Thejas Nair
[jira] [Updated] (HIVE-4373) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect
[ https://issues.apache.org/jira/browse/HIVE-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-4373: Release Note: rb link - https://reviews.apache.org/r/10812/ Status: Patch Available (was: Open) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect Key: HIVE-4373 URL: https://issues.apache.org/jira/browse/HIVE-4373 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Reporter: Deepesh Khandelwal Assignee: Thejas M Nair Priority: Minor Fix For: 0.11.0 Attachments: HIVE-4373.1.patch When running beeline {code} % beeline -u 'jdbc:hive2://localhost:1' -n hive -p passwd -d org.apache.hive.jdbc.HiveDriver Connecting to jdbc:hive2://localhost:1 Connected to: Hive (version 0.10.0) Driver: Hive (version 0.11.0) Transaction isolation: TRANSACTION_REPEATABLE_READ {code} The Hive version in the Connected to: string says 0.10.0 instead of 0.11.0. Looking at the code it seems that the version is hardcoded at two places: line 250 in jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java line 833 in jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Jenkins build is back to normal : Hive-0.9.1-SNAPSHOT-h0.21 #359
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/359/
[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function
[ https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643546#comment-13643546 ] Lefty Leverenz commented on HIVE-4019: -- [Taking this off the JIRA] Thanks for the suggestion, Harish. I gave it a try, but found out we can't use semicolons or backslashes: So I abandoned the ampersand and spelled out WindowingAndAnalytics. The system fixed the link on the Language Manual page automatically. Now following the link gives the right URL: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics Good enough. – Lefty On Fri, Apr 26, 2013 at 11:42 AM, Harish Butani (JIRA) j...@apache.orgwrote: Ability to create and drop temporary partition function --- Key: HIVE-4019 URL: https://issues.apache.org/jira/browse/HIVE-4019 Project: Hive Issue Type: New Feature Components: PTF-Windowing Reporter: Ashutosh Chauhan Assignee: Brock Noland Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, hive-4019.q Just like udf/udaf/udtf functions, user should be able to add and drop custom partitioning functions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function
[ https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643548#comment-13643548 ] Lefty Leverenz commented on HIVE-4019: -- Hm, so replying to an email thread doesn't take a conversation off the JIRA. Live learn. Ability to create and drop temporary partition function --- Key: HIVE-4019 URL: https://issues.apache.org/jira/browse/HIVE-4019 Project: Hive Issue Type: New Feature Components: PTF-Windowing Reporter: Ashutosh Chauhan Assignee: Brock Noland Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, hive-4019.q Just like udf/udaf/udtf functions, user should be able to add and drop custom partitioning functions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira