[jira] [Assigned] (HIVE-3566) Hive List Bucketing - Query Logic Optimization
[ https://issues.apache.org/jira/browse/HIVE-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu reassigned HIVE-3566: -- Assignee: Gang Tim Liu Hive List Bucketing - Query Logic Optimization -- Key: HIVE-3566 URL: https://issues.apache.org/jira/browse/HIVE-3566 Project: Hive Issue Type: Improvement Reporter: Gang Tim Liu Assignee: Gang Tim Liu Priority: Minor There are a few places to optimize list bucketing query: 1. evaluateExprOnCell() need not be called for all cells for 'Other'. Stop the first time, any 'Other' is false. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3566) Hive List Bucketing - Query Logic Optimization
Gang Tim Liu created HIVE-3566: -- Summary: Hive List Bucketing - Query Logic Optimization Key: HIVE-3566 URL: https://issues.apache.org/jira/browse/HIVE-3566 Project: Hive Issue Type: Improvement Reporter: Gang Tim Liu Priority: Minor There are a few places to optimize list bucketing query: 1. evaluateExprOnCell() need not be called for all cells for 'Other'. Stop the first time, any 'Other' is false. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3567) Hive List Bucketing - Query logic supports more operator thank equal
Gang Tim Liu created HIVE-3567: -- Summary: Hive List Bucketing - Query logic supports more operator thank equal Key: HIVE-3567 URL: https://issues.apache.org/jira/browse/HIVE-3567 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Query logic supports equal. It should handle , =, and = also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3567) Hive List Bucketing - Query logic supports more operator thank equal
[ https://issues.apache.org/jira/browse/HIVE-3567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu reassigned HIVE-3567: -- Assignee: Gang Tim Liu Hive List Bucketing - Query logic supports more operator thank equal Key: HIVE-3567 URL: https://issues.apache.org/jira/browse/HIVE-3567 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Assignee: Gang Tim Liu Query logic supports equal. It should handle , =, and = also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3567) Hive List Bucketing - Query logic supports more operator than equal
[ https://issues.apache.org/jira/browse/HIVE-3567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3567: --- Summary: Hive List Bucketing - Query logic supports more operator than equal (was: Hive List Bucketing - Query logic supports more operator thank equal) Hive List Bucketing - Query logic supports more operator than equal --- Key: HIVE-3567 URL: https://issues.apache.org/jira/browse/HIVE-3567 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Assignee: Gang Tim Liu Query logic supports equal. It should handle , =, and = also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3276) optimize union sub-queries
[ https://issues.apache.org/jira/browse/HIVE-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473935#comment-13473935 ] Carl Steinbach commented on HIVE-3276: -- @Namit: I added two comments on phabricator. I'm looking at this pretty late so feel free to ignore them. optimize union sub-queries -- Key: HIVE-3276 URL: https://issues.apache.org/jira/browse/HIVE-3276 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3276.10.patch, hive.3276.11.patch, HIVE-3276.1.patch, hive.3276.2.patch, hive.3276.3.patch, hive.3276.4.patch, hive.3276.5.patch, hive.3276.6.patch, hive.3276.7.patch, hive.3276.8.patch, hive.3276.9.patch It might be a good idea to optimize simple union queries containing map-reduce jobs in at least one of the sub-qeuries. For eg: a query like: insert overwrite table T1 partition P1 select * from ( subq1 union all subq2 ) u; today creates 3 map-reduce jobs, one for subq1, another for subq2 and the final one for the union. It might be a good idea to optimize this. Instead of creating the union task, it might be simpler to create a move task (or something like a move task), where the outputs of the two sub-queries will be moved to the final directory. This can easily extend to more than 2 sub-queries in the union. This is very useful if there is a select * followed by filesink after the union. This can be independently useful, and also be used to optimize the skewed joins https://cwiki.apache.org/Hive/skewed-join-optimization.html. If there is a select, filter between the union and the filesink, the select and the filter can be moved before the union, and the follow-up job can still be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3568) No pushdown subquery's group by key in outside filter
Gang Tim Liu created HIVE-3568: -- Summary: No pushdown subquery's group by key in outside filter Key: HIVE-3568 URL: https://issues.apache.org/jira/browse/HIVE-3568 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Gang Tim Liu Priority: Minor pushdown doesn't happen with group by in subquery and group by key in outside filter. select * from (select x, count(1) from fact_daily where ds ='1' group by x) subq where x = 484; subquery still scans the partition. expect only scan x = 484. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3554) Hive List Bucketing - Query logic
[ https://issues.apache.org/jira/browse/HIVE-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3554: --- Status: Patch Available (was: In Progress) Yes, link other issues. Yes, patch is available. Hive List Bucketing - Query logic - Key: HIVE-3554 URL: https://issues.apache.org/jira/browse/HIVE-3554 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3554.patch.1, HIVE-3554.patch.2, HIVE-3554.patch.3 This is part of efforts for list bucketing feature: https://cwiki.apache.org/Hive/listbucketing.html This patch includes: 1. Query logic: hive chooses right sub-directory instead of partition directory. 2. alter table grammar which is required to support query logic This patch doesn't include list bucketing DML. Main reasons: 1. risk. w/o DML, this patch won't impact any existing hive regression features since no touch on any data manipulation so that very low risk. 2. manageability. w/ DML, patch is getting bigger and hard to review. Removing DML, it's easy to review. We still disable hive feature by default since DML is not in yet. DML will be in follow-up patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3554) Hive List Bucketing - Query logic
[ https://issues.apache.org/jira/browse/HIVE-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3554: --- Attachment: HIVE-3554.patch.3 Hive List Bucketing - Query logic - Key: HIVE-3554 URL: https://issues.apache.org/jira/browse/HIVE-3554 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3554.patch.1, HIVE-3554.patch.2, HIVE-3554.patch.3 This is part of efforts for list bucketing feature: https://cwiki.apache.org/Hive/listbucketing.html This patch includes: 1. Query logic: hive chooses right sub-directory instead of partition directory. 2. alter table grammar which is required to support query logic This patch doesn't include list bucketing DML. Main reasons: 1. risk. w/o DML, this patch won't impact any existing hive regression features since no touch on any data manipulation so that very low risk. 2. manageability. w/ DML, patch is getting bigger and hard to review. Removing DML, it's easy to review. We still disable hive feature by default since DML is not in yet. DML will be in follow-up patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-3554) Hive List Bucketing - Query logic
[ https://issues.apache.org/jira/browse/HIVE-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-3554 started by Gang Tim Liu. Hive List Bucketing - Query logic - Key: HIVE-3554 URL: https://issues.apache.org/jira/browse/HIVE-3554 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3554.patch.1, HIVE-3554.patch.2, HIVE-3554.patch.3 This is part of efforts for list bucketing feature: https://cwiki.apache.org/Hive/listbucketing.html This patch includes: 1. Query logic: hive chooses right sub-directory instead of partition directory. 2. alter table grammar which is required to support query logic This patch doesn't include list bucketing DML. Main reasons: 1. risk. w/o DML, this patch won't impact any existing hive regression features since no touch on any data manipulation so that very low risk. 2. manageability. w/ DML, patch is getting bigger and hard to review. Removing DML, it's easy to review. We still disable hive feature by default since DML is not in yet. DML will be in follow-up patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3213) ODBC API enhancements
[ https://issues.apache.org/jira/browse/HIVE-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473969#comment-13473969 ] Carl Steinbach commented on HIVE-3213: -- @Prasad: Please attach the patch, provide a link to the review request, and change the status to 'patch available' when you're ready to have this reviewed. Thanks. ODBC API enhancements - Key: HIVE-3213 URL: https://issues.apache.org/jira/browse/HIVE-3213 Project: Hive Issue Type: Sub-task Components: ODBC Affects Versions: 0.10.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.10.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #165
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/ -- [...truncated 10125 lines...] [echo] Project: odbc [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/odbc/src/conf does not exist. ivy-resolve-test: [echo] Project: odbc ivy-retrieve-test: [echo] Project: odbc compile-test: [echo] Project: odbc create-dirs: [echo] Project: serde [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/serde/src/test/resources does not exist. init: [echo] Project: serde ivy-init-settings: [echo] Project: serde ivy-resolve: [echo] Project: serde [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-serde-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/ivy/report/org.apache.hive-hive-serde-default.html ivy-retrieve: [echo] Project: serde dynamic-serde: compile: [echo] Project: serde ivy-resolve-test: [echo] Project: serde ivy-retrieve-test: [echo] Project: serde compile-test: [echo] Project: serde [javac] Compiling 26 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/serde/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. create-dirs: [echo] Project: service [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/service/src/test/resources does not exist. init: [echo] Project: service ivy-init-settings: [echo] Project: service ivy-resolve: [echo] Project: service [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-service-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/ivy/report/org.apache.hive-hive-service-default.html ivy-retrieve: [echo] Project: service compile: [echo] Project: service ivy-resolve-test: [echo] Project: service ivy-retrieve-test: [echo] Project: service compile-test: [echo] Project: service [javac] Compiling 2 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/service/test/classes test: [echo] Project: hive test-shims: [echo] Project: hive test-conditions: [echo] Project: shims gen-test: [echo] Project: shims create-dirs: [echo] Project: shims [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/test/resources does not exist. init: [echo] Project: shims ivy-init-settings: [echo] Project: shims ivy-resolve: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-shims-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/ivy/report/org.apache.hive-hive-shims-default.html ivy-retrieve: [echo] Project: shims compile: [echo] Project: shims [echo] Building shims 0.20 build_shims: [echo] Project: shims [echo] Compiling https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.20/java against hadoop 0.20.2 (https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/165/artifact/hive/build/hadoopcore/hadoop-0.20.2) ivy-init-settings: [echo] Project: shims ivy-resolve-hadoop-shim: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml ivy-retrieve-hadoop-shim: [echo] Project: shims [echo] Building shims 0.20S build_shims: [echo] Project: shims [echo] Compiling
[jira] [Updated] (HIVE-3554) Hive List Bucketing - Query logic
[ https://issues.apache.org/jira/browse/HIVE-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3554: --- Attachment: HIVE-3554.patch.4 Hive List Bucketing - Query logic - Key: HIVE-3554 URL: https://issues.apache.org/jira/browse/HIVE-3554 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3554.patch.1, HIVE-3554.patch.2, HIVE-3554.patch.3, HIVE-3554.patch.4 This is part of efforts for list bucketing feature: https://cwiki.apache.org/Hive/listbucketing.html This patch includes: 1. Query logic: hive chooses right sub-directory instead of partition directory. 2. alter table grammar which is required to support query logic This patch doesn't include list bucketing DML. Main reasons: 1. risk. w/o DML, this patch won't impact any existing hive regression features since no touch on any data manipulation so that very low risk. 2. manageability. w/ DML, patch is getting bigger and hard to review. Removing DML, it's easy to review. We still disable hive feature by default since DML is not in yet. DML will be in follow-up patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3569) RCFile requires native Hadoop library
[
https://issues.apache.org/jira/browse/HIVE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Phillips updated HIVE-3569:
-
Description:
RCFile requires the native Hadoop library. It does not work when using the Java
{{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java {{GZIPInputStream}},
which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the stream is still empty at that
point).
{noformat}
12/10/11 10:37:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
Exception in thread main java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:254)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:163)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:78)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:90)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.init(GzipCodec.java:92)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.init(GzipCodec.java:101)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:169)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:179)
at
org.apache.hadoop.hive.ql.io.RCFile$ValueBuffer.init(RCFile.java:451)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1205)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:)
at
org.apache.hadoop.hive.ql.io.RCFileRecordReader.init(RCFileRecordReader.java:52)
{noformat}
was:
RCFile requires the native Hadoop library. It does not work when using the Java
{{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java {{GZIPInputStream}},
which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the stream is still empty at that
point).
{noformat}
12/10/11 10:37:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
Exception in thread main java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:254)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:163)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:78)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:90)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.init(GzipCodec.java:92)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.init(GzipCodec.java:101)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:169)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:179)
at
org.apache.hadoop.hive.ql.io.RCFile$ValueBuffer.init(RCFile.java:451)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1205)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:)
at
org.apache.hadoop.hive.ql.io.RCFileRecordReader.init(RCFileRecordReader.java:52)
{noformat}
RCFile requires native Hadoop library
-
Key: HIVE-3569
URL: https://issues.apache.org/jira/browse/HIVE-3569
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: David Phillips
RCFile requires the native Hadoop library. It does not work when using the
Java {{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java
{{GZIPInputStream}}, which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the
[jira] [Updated] (HIVE-3569) RCFile requires native Hadoop library
[
https://issues.apache.org/jira/browse/HIVE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Phillips updated HIVE-3569:
-
Description:
RCFile requires the native Hadoop library. It does not work when using the Java
{{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java {{GZIPInputStream}},
which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the stream is still empty at that
point).
{noformat}
12/10/11 10:37:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
Exception in thread main java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:254)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:163)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:78)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:90)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.init(GzipCodec.java:92)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.init(GzipCodec.java:101)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:169)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:179)
at
org.apache.hadoop.hive.ql.io.RCFile$ValueBuffer.init(RCFile.java:451)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1205)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:)
at
org.apache.hadoop.hive.ql.io.RCFileRecordReader.init(RCFileRecordReader.java:52)
{noformat}
was:
RCFile requires the native Hadoop library. It does not work when using the Java
{{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java {{GZIPInputStream}},
which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the stream is still empty at that
point).
{noformat}
12/10/11 10:37:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
Exception in thread main java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:254)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:163)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:78)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:90)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.init(GzipCodec.java:92)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.init(GzipCodec.java:101)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:169)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:179)
at
org.apache.hadoop.hive.ql.io.RCFile$ValueBuffer.init(RCFile.java:451)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1205)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:)
at
org.apache.hadoop.hive.ql.io.RCFileRecordReader.init(RCFileRecordReader.java:52)
{noformat}
RCFile requires native Hadoop library
-
Key: HIVE-3569
URL: https://issues.apache.org/jira/browse/HIVE-3569
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: David Phillips
RCFile requires the native Hadoop library. It does not work when using the
Java {{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java
{{GZIPInputStream}}, which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the
[jira] [Updated] (HIVE-3569) RCFile requires native Hadoop library
[
https://issues.apache.org/jira/browse/HIVE-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Phillips updated HIVE-3569:
-
Description:
RCFile requires the native Hadoop library. It does not work when using the Java
{{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java {{GZIPInputStream}},
which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the stream is still empty at that
point).
{noformat}
12/10/11 10:37:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
Exception in thread main java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:254)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:163)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:78)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:90)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.init(GzipCodec.java:92)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.init(GzipCodec.java:101)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:169)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:179)
at
org.apache.hadoop.hive.ql.io.RCFile$ValueBuffer.init(RCFile.java:451)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1205)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:)
at
org.apache.hadoop.hive.ql.io.RCFileRecordReader.init(RCFileRecordReader.java:52)
{noformat}
was:
RCFile requires the native Hadoop library. It does not work when using the Java
{{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java {{GZIPInputStream}},
which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the stream is still empty at that
point).
{noformat}
12/10/11 10:37:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
12/10/11 10:37:25 INFO io.CodecPool: Got brand-new decompressor
Exception in thread main java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:254)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:163)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:78)
at java.util.zip.GZIPInputStream.init(GZIPInputStream.java:90)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.init(GzipCodec.java:92)
at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.init(GzipCodec.java:101)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:169)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:179)
at
org.apache.hadoop.hive.ql.io.RCFile$ValueBuffer.init(RCFile.java:451)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1205)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:)
at
org.apache.hadoop.hive.ql.io.RCFileRecordReader.init(RCFileRecordReader.java:52)
{noformat}
RCFile requires native Hadoop library
-
Key: HIVE-3569
URL: https://issues.apache.org/jira/browse/HIVE-3569
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: David Phillips
RCFile requires the native Hadoop library. It does not work when using the
Java {{GzipCodec}}.
The root cause is that the two versions of {{GzipCodec.createInputStream()}}
work differently. The native version simply saves a reference to the supplied
input stream. The Java version wraps the stream in a Java
{{GZIPInputStream}}, which immediately tries to read the header.
The problem occurs because the stream passed by the {{RCFile.ValueBuffer}}
constructor is empty (the buffer backing the
[jira] [Created] (HIVE-3570) Add/fix facility to collect operator specific statisticsin hive + add hash-in/hash-out counter for GroupBy Optr
Satadru Pan created HIVE-3570: - Summary: Add/fix facility to collect operator specific statisticsin hive + add hash-in/hash-out counter for GroupBy Optr Key: HIVE-3570 URL: https://issues.apache.org/jira/browse/HIVE-3570 Project: Hive Issue Type: Improvement Components: Statistics Affects Versions: 0.9.0 Reporter: Satadru Pan Priority: Minor Requirement: Collect Operator specific stats for hive queries. Use the counter framework available in Hive Operator.java to accomplish that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3570) Add/fix facility to collect operator specific statisticsin hive + add hash-in/hash-out counter for GroupBy Optr
[ https://issues.apache.org/jira/browse/HIVE-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satadru Pan updated HIVE-3570: -- Attachment: HIVE-3570.1.patch.txt Patch for the task HIVE-3570 Add/fix facility to collect operator specific statisticsin hive + add hash-in/hash-out counter for GroupBy Optr --- Key: HIVE-3570 URL: https://issues.apache.org/jira/browse/HIVE-3570 Project: Hive Issue Type: Improvement Components: Statistics Affects Versions: 0.9.0 Reporter: Satadru Pan Priority: Minor Attachments: HIVE-3570.1.patch.txt Requirement: Collect Operator specific stats for hive queries. Use the counter framework available in Hive Operator.java to accomplish that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3564) hivetest.py: revision number and applied patch
[ https://issues.apache.org/jira/browse/HIVE-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Gorbachev updated HIVE-3564: - Status: Patch Available (was: Open) hivetest.py: revision number and applied patch -- Key: HIVE-3564 URL: https://issues.apache.org/jira/browse/HIVE-3564 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Ivan Gorbachev Assignee: Ivan Gorbachev It's required to add new option for hivetest.py which will allow to show base revision number and applied patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3564) hivetest.py: revision number and applied patch
[ https://issues.apache.org/jira/browse/HIVE-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474475#comment-13474475 ] Ivan Gorbachev commented on HIVE-3564: -- https://reviews.facebook.net/D5973 hivetest.py: revision number and applied patch -- Key: HIVE-3564 URL: https://issues.apache.org/jira/browse/HIVE-3564 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Ivan Gorbachev Assignee: Ivan Gorbachev Attachments: hive-3564.0.patch.txt It's required to add new option for hivetest.py which will allow to show base revision number and applied patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3564) hivetest.py: revision number and applied patch
[ https://issues.apache.org/jira/browse/HIVE-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Gorbachev updated HIVE-3564: - Attachment: hive-3564.0.patch.txt hivetest.py: revision number and applied patch -- Key: HIVE-3564 URL: https://issues.apache.org/jira/browse/HIVE-3564 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Ivan Gorbachev Assignee: Ivan Gorbachev Attachments: hive-3564.0.patch.txt It's required to add new option for hivetest.py which will allow to show base revision number and applied patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3570) Add/fix facility to collect operator specific statisticsin hive + add hash-in/hash-out counter for GroupBy Optr
[ https://issues.apache.org/jira/browse/HIVE-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain reassigned HIVE-3570: Assignee: Satadru Pan Add/fix facility to collect operator specific statisticsin hive + add hash-in/hash-out counter for GroupBy Optr --- Key: HIVE-3570 URL: https://issues.apache.org/jira/browse/HIVE-3570 Project: Hive Issue Type: Improvement Components: Statistics Affects Versions: 0.9.0 Reporter: Satadru Pan Assignee: Satadru Pan Priority: Minor Attachments: HIVE-3570.1.patch.txt Requirement: Collect Operator specific stats for hive queries. Use the counter framework available in Hive Operator.java to accomplish that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3564) hivetest.py: revision number and applied patch
[ https://issues.apache.org/jira/browse/HIVE-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474520#comment-13474520 ] Namit Jain commented on HIVE-3564: -- +1 hivetest.py: revision number and applied patch -- Key: HIVE-3564 URL: https://issues.apache.org/jira/browse/HIVE-3564 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Ivan Gorbachev Assignee: Ivan Gorbachev Attachments: hive-3564.0.patch.txt It's required to add new option for hivetest.py which will allow to show base revision number and applied patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-446) Implement TRUNCATE
[ https://issues.apache.org/jira/browse/HIVE-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Chalfant reassigned HIVE-446: Assignee: Andrew Chalfant Implement TRUNCATE -- Key: HIVE-446 URL: https://issues.apache.org/jira/browse/HIVE-446 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Prasad Chakka Assignee: Andrew Chalfant truncate the data but leave the table and metadata intact. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1733 - Still Failing
Changes for Build #1728 Changes for Build #1729 Changes for Build #1730 [namit] HIVE-3550 Early skipping for limit operator at reduce stage (Navis via namit) [namit] HIVE-3536 Output of sort merge join is no longer bucketed (Kevin Wilfong) For some reason, I missed merging the test in my previous merge [kevinwilfong] HIVE-3522. Make separator for Entity name configurable. (Raghotham Murthy via kevinwilfong) Changes for Build #1731 Changes for Build #1732 Changes for Build #1733 1 tests failed. FAILED: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_aggregator_error_1 Error Message: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit. Stack Trace: junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit. at net.sf.antcontrib.logic.ForTask.doSequentialIteration(ForTask.java:259) at net.sf.antcontrib.logic.ForTask.doToken(ForTask.java:268) at net.sf.antcontrib.logic.ForTask.doTheTasks(ForTask.java:324) at net.sf.antcontrib.logic.ForTask.execute(ForTask.java:244) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1733) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1733/ to view the results.
[jira] [Created] (HIVE-3571) add a way to run a small unit quickly
Namit Jain created HIVE-3571: Summary: add a way to run a small unit quickly Key: HIVE-3571 URL: https://issues.apache.org/jira/browse/HIVE-3571 Project: Hive Issue Type: Test Components: Testing Infrastructure Reporter: Namit Jain A simple unit test: ant test -Dtestcase=TestCliDriver -Dqfile=groupby2.q takes a long time. There should be a quick way to achieve that for debugging. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3518) QTestUtil side-effects
[
https://issues.apache.org/jira/browse/HIVE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3518:
Assignee: Navis
Status: Patch Available (was: Open)
QTestUtil side-effects
--
Key: HIVE-3518
URL: https://issues.apache.org/jira/browse/HIVE-3518
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure, Tests
Reporter: Ivan Gorbachev
Assignee: Navis
Attachments: HIVE-3518.D5865.1.patch, HIVE-3518.D5865.2.patch,
metadata_export_drop.q
It seems that QTestUtil has side-effects. This test
([^metadata_export_drop.q]) causes failure of other tests on cleanup stage:
{quote}
Exception: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:845)
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:821)
at org.apache.hadoop.hive.ql.QTestUtil.cleanUp(QTestUtil.java:445)
at org.apache.hadoop.hive.ql.QTestUtil.shutdown(QTestUtil.java:300)
at org.apache.hadoop.hive.cli.TestCliDriver.tearDown(TestCliDriver.java:87)
at junit.framework.TestCase.runBare(TestCase.java:140)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
at org.apache.hadoop.fs.Path.initialize(Path.java:140)
at org.apache.hadoop.fs.Path.init(Path.java:132)
at
org.apache.hadoop.fs.ProxyFileSystem.swizzleParamPath(ProxyFileSystem.java:56)
at org.apache.hadoop.fs.ProxyFileSystem.mkdirs(ProxyFileSystem.java:214)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1120)
at
org.apache.hadoop.hive.ql.parse.MetaDataExportListener.export_meta_data(MetaDataExportListener.java:81)
at
org.apache.hadoop.hive.ql.parse.MetaDataExportListener.onEvent(MetaDataExportListener.java:106)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1024)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:1185)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:566)
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:839)
... 17 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
at java.net.URI.checkPath(URI.java:1787)
at java.net.URI.init(URI.java:735)
at org.apache.hadoop.fs.Path.initialize(Path.java:137)
... 28 more
{quote}
Flushing 'hive.metastore.pre.event.listeners' into empty string solves the
issue. During debugging I figured out this property wan't cleaned for other
tests after it was set in metadata_export_drop.q.
How to reproduce:
{code} ant test -Dtestcase=TestCliDriver -Dqfile=metadata_export_drop.q,some
test.q{code}
where some test.q means any test which contains CREATE statement. For
example, sample10.q
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3518) QTestUtil side-effects
[
https://issues.apache.org/jira/browse/HIVE-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3518:
--
Attachment: HIVE-3518.D5865.2.patch
navis updated the revision HIVE-3518 [jira] QTestUtil side-effects.
Reviewers: JIRA
Resetting conf made other issues, so just sure new conf propagated to meta
store.
REVISION DETAIL
https://reviews.facebook.net/D5865
AFFECTED FILES
ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
To: JIRA, navis
QTestUtil side-effects
--
Key: HIVE-3518
URL: https://issues.apache.org/jira/browse/HIVE-3518
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure, Tests
Reporter: Ivan Gorbachev
Assignee: Navis
Attachments: HIVE-3518.D5865.1.patch, HIVE-3518.D5865.2.patch,
metadata_export_drop.q
It seems that QTestUtil has side-effects. This test
([^metadata_export_drop.q]) causes failure of other tests on cleanup stage:
{quote}
Exception: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:845)
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:821)
at org.apache.hadoop.hive.ql.QTestUtil.cleanUp(QTestUtil.java:445)
at org.apache.hadoop.hive.ql.QTestUtil.shutdown(QTestUtil.java:300)
at org.apache.hadoop.hive.cli.TestCliDriver.tearDown(TestCliDriver.java:87)
at junit.framework.TestCase.runBare(TestCase.java:140)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
at org.apache.hadoop.fs.Path.initialize(Path.java:140)
at org.apache.hadoop.fs.Path.init(Path.java:132)
at
org.apache.hadoop.fs.ProxyFileSystem.swizzleParamPath(ProxyFileSystem.java:56)
at org.apache.hadoop.fs.ProxyFileSystem.mkdirs(ProxyFileSystem.java:214)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1120)
at
org.apache.hadoop.hive.ql.parse.MetaDataExportListener.export_meta_data(MetaDataExportListener.java:81)
at
org.apache.hadoop.hive.ql.parse.MetaDataExportListener.onEvent(MetaDataExportListener.java:106)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1024)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:1185)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:566)
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:839)
... 17 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI:
file:../build/ql/test/data/exports/HIVE-3427/src.2012-09-28-11-38-17
at java.net.URI.checkPath(URI.java:1787)
at java.net.URI.init(URI.java:735)
at org.apache.hadoop.fs.Path.initialize(Path.java:137)
... 28 more
{quote}
Flushing 'hive.metastore.pre.event.listeners' into empty string solves the
issue. During debugging I figured out this property wan't cleaned for other
tests after it was set in metadata_export_drop.q.
How to reproduce:
{code} ant test -Dtestcase=TestCliDriver -Dqfile=metadata_export_drop.q,some
test.q{code}
where some test.q means any test which contains CREATE statement. For
example, sample10.q
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3554) Hive List Bucketing - Query logic
[ https://issues.apache.org/jira/browse/HIVE-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3554: --- Attachment: HIVE-3554.patch.5 Hive List Bucketing - Query logic - Key: HIVE-3554 URL: https://issues.apache.org/jira/browse/HIVE-3554 Project: Hive Issue Type: New Feature Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3554.patch.1, HIVE-3554.patch.2, HIVE-3554.patch.3, HIVE-3554.patch.4, HIVE-3554.patch.5 This is part of efforts for list bucketing feature: https://cwiki.apache.org/Hive/listbucketing.html This patch includes: 1. Query logic: hive chooses right sub-directory instead of partition directory. 2. alter table grammar which is required to support query logic This patch doesn't include list bucketing DML. Main reasons: 1. risk. w/o DML, this patch won't impact any existing hive regression features since no touch on any data manipulation so that very low risk. 2. manageability. w/ DML, patch is getting bigger and hard to review. Removing DML, it's easy to review. We still disable hive feature by default since DML is not in yet. DML will be in follow-up patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3377) ant model-jar command fails in metastore
[ https://issues.apache.org/jira/browse/HIVE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474763#comment-13474763 ] Kevin Wilfong commented on HIVE-3377: - Deleting line 112 in build-common.xml seems to help. I don't know if the build will work in all operating systems without it though. ant model-jar command fails in metastore Key: HIVE-3377 URL: https://issues.apache.org/jira/browse/HIVE-3377 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0 Reporter: Vandana Ayyalasomayajula Priority: Minor Labels: build Running ant model-jar command to set up eclipse dev environment from the following wiki: https://cwiki.apache.org/Hive/gettingstarted-eclipsesetup.html fails with the following message: BUILD FAILED **/workspace/hive-trunk/metastore/build.xml:22: The following error occurred while executing this line: **/workspace/hive-trunk/build-common.xml:112: Problem: failed to create task or type osfamily Cause: The name is undefined. Action: Check the spelling. Action: Check that any custom tasks/types have been declared. Action: Check that any presetdef/macrodef declarations have taken place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3377) ant model-jar command fails in metastore
[
https://issues.apache.org/jira/browse/HIVE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474768#comment-13474768
]
Krish commented on HIVE-3377:
-
Line # 112 is blank; here are line numbers 105 - 120. please let me know which
line to delete.
105 !-- include contrib on local classpath, but not on cluster --
106 !-- https://reviews.facebook.net/D2133#comment-47 --
107 path id=test.local.classpath
108 path refid=${test.classpath.id}/
109 fileset dir=${hive.root}/build/ivy/lib/test
includes=hive-contrib*.jar erroronmissingdir=false/
110 /path
111
112
113 loadproperties srcfile=${ivy.conf.dir}/libraries.properties/
114
115 osfamily property=os.family/
116
117 condition property=offline
118 istrue value=${is-offline}/
119 /condition
120 import file=build-offline.xml/
ant model-jar command fails in metastore
Key: HIVE-3377
URL: https://issues.apache.org/jira/browse/HIVE-3377
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.10.0
Reporter: Vandana Ayyalasomayajula
Priority: Minor
Labels: build
Running ant model-jar command to set up eclipse dev environment from the
following wiki:
https://cwiki.apache.org/Hive/gettingstarted-eclipsesetup.html
fails with the following message:
BUILD FAILED
**/workspace/hive-trunk/metastore/build.xml:22: The following error occurred
while executing this line:
**/workspace/hive-trunk/build-common.xml:112: Problem: failed to create task
or type osfamily
Cause: The name is undefined.
Action: Check the spelling.
Action: Check that any custom tasks/types have been declared.
Action: Check that any presetdef/macrodef declarations have taken place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3377) ant model-jar command fails in metastore
[ https://issues.apache.org/jira/browse/HIVE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474773#comment-13474773 ] Kevin Wilfong commented on HIVE-3377: - 115 then ant model-jar command fails in metastore Key: HIVE-3377 URL: https://issues.apache.org/jira/browse/HIVE-3377 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0 Reporter: Vandana Ayyalasomayajula Priority: Minor Labels: build Running ant model-jar command to set up eclipse dev environment from the following wiki: https://cwiki.apache.org/Hive/gettingstarted-eclipsesetup.html fails with the following message: BUILD FAILED **/workspace/hive-trunk/metastore/build.xml:22: The following error occurred while executing this line: **/workspace/hive-trunk/build-common.xml:112: Problem: failed to create task or type osfamily Cause: The name is undefined. Action: Check the spelling. Action: Check that any custom tasks/types have been declared. Action: Check that any presetdef/macrodef declarations have taken place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3377) ant model-jar command fails in metastore
[ https://issues.apache.org/jira/browse/HIVE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474782#comment-13474782 ] Krish commented on HIVE-3377: - Thanks, that worked but next command ant gen-test failed. Here is the error message, any thoughts? /opt/hive-trunk$ ant get-test Buildfile: /opt/hive-trunk/build.xml BUILD FAILED Target get-test does not exist in the project hive. Total time: 1 second ant model-jar command fails in metastore Key: HIVE-3377 URL: https://issues.apache.org/jira/browse/HIVE-3377 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0 Reporter: Vandana Ayyalasomayajula Priority: Minor Labels: build Running ant model-jar command to set up eclipse dev environment from the following wiki: https://cwiki.apache.org/Hive/gettingstarted-eclipsesetup.html fails with the following message: BUILD FAILED **/workspace/hive-trunk/metastore/build.xml:22: The following error occurred while executing this line: **/workspace/hive-trunk/build-common.xml:112: Problem: failed to create task or type osfamily Cause: The name is undefined. Action: Check the spelling. Action: Check that any custom tasks/types have been declared. Action: Check that any presetdef/macrodef declarations have taken place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3377) ant model-jar command fails in metastore
[ https://issues.apache.org/jira/browse/HIVE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474787#comment-13474787 ] Kevin Wilfong commented on HIVE-3377: - Sorry, that one's new to me. ant model-jar command fails in metastore Key: HIVE-3377 URL: https://issues.apache.org/jira/browse/HIVE-3377 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0 Reporter: Vandana Ayyalasomayajula Priority: Minor Labels: build Running ant model-jar command to set up eclipse dev environment from the following wiki: https://cwiki.apache.org/Hive/gettingstarted-eclipsesetup.html fails with the following message: BUILD FAILED **/workspace/hive-trunk/metastore/build.xml:22: The following error occurred while executing this line: **/workspace/hive-trunk/build-common.xml:112: Problem: failed to create task or type osfamily Cause: The name is undefined. Action: Check the spelling. Action: Check that any custom tasks/types have been declared. Action: Check that any presetdef/macrodef declarations have taken place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
