Re: Cloning Hive Trunk: HadoopArchives Dependency Not Found
Hmm, so I just ran the same command with 'mvn clean compile -Phadoop-1' and
'mvn clean compile -Phadoop-2' and they seem to work.
Thanks @all! I guess this wasn't an issue. :)
Thanks and Regards,
Nishant Kelkar
On Wed, Jun 24, 2015 at 1:59 AM, Nishant Kelkar nishant@gmail.com
wrote:
Hi all,
So I am a beginner to Hive, pardon me if I'm doing something wrong. I
recently cloned the Hive dev trunk (master branch) via Git. I then went
into my IDE (IntelliJ) and started looking around. At first, it resolved a
lot of dependencies. However, after my IDE finished pulling from different
sources, I saw that it was still missing the *HadoopArchives* dependency.
Later on, I was trying to build the hive/ql sub-project with 'mvn clean
compile', but I'm getting the following error:
*---BEGIN---*
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
(default-compile) on project hive-exec: Compilation failure: Compilation
failure:
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:[213,31]
cannot find symbol
[ERROR] symbol: class HadoopArchives
[ERROR] location: package org.apache.hadoop.tools
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java:[42,41]
cannot find symbol
[ERROR] symbol: class TimelineClient
[ERROR] location: package org.apache.hadoop.yarn.client.api
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java:[57,18]
cannot find symbol
[ERROR] symbol: class TimelineClient
[ERROR] location: class org.apache.hadoop.hive.ql.hooks.ATSHook
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:[1351,9]
cannot find symbol
[ERROR] symbol: class HadoopArchives
[ERROR] location: class org.apache.hadoop.hive.ql.exec.DDLTask
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:[1351,34]
cannot find symbol
[ERROR] symbol: class HadoopArchives
[ERROR] location: class org.apache.hadoop.hive.ql.exec.DDLTask
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java:[72,26]
cannot find symbol
[ERROR] symbol: variable TimelineClient
[ERROR] location: class org.apache.hadoop.hive.ql.hooks.ATSHook
[ERROR] - [Help 1]
*---END---*
I looked at the pom.xml, and the following is highlighted in red:
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-archives/artifactId
version${hadoop-23.version}/version
optionaltrue/optional
/dependency
This basically means that IntelliJ/Maven was not able to find the
dependency under the Maven repo. I saw the ticket HIVE-9048
https://issues.apache.org/jira/browse/HIVE-9048 and HadoopArchives
should be a sub-dependency of the hadoop-tools dependency right?
Here's a copy of the relevant mirrors that I'm using in my settings.xml:
mirror
idConjars/id
mirrorOf*/mirrorOf
nameUS Conjars/name
urlhttp://conjars.org/repo/url
/mirror
mirror
idUS/id
mirrorOfcentral/mirrorOf
nameUS Central/name
urlhttp://repo.maven.apache.org/maven2/url
/mirror
Does anyone have an idea why this could be happening? I was kind of
expecting just cloning from Git, and running 'mvn clean compile' and seeing
everything pass and build with success.
Best Regards,
Nishant Kelkar
Re: Cloning Hive Trunk: HadoopArchives Dependency Not Found
Hi all,
So I am a beginner to Hive, pardon me if I'm doing something wrong. I
recently cloned the Hive dev trunk (master branch) via Git. I then went
into my IDE (IntelliJ) and started looking around. At first, it resolved a
lot of dependencies. However, after my IDE finished pulling from different
sources, I saw that it was still missing the *HadoopArchives* dependency.
Later on, I was trying to build the hive/ql sub-project with 'mvn clean
compile', but I'm getting the following error:
*---BEGIN---*
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
(default-compile) on project hive-exec: Compilation failure: Compilation
failure:
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:[213,31]
cannot find symbol
[ERROR] symbol: class HadoopArchives
[ERROR] location: package org.apache.hadoop.tools
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java:[42,41]
cannot find symbol
[ERROR] symbol: class TimelineClient
[ERROR] location: package org.apache.hadoop.yarn.client.api
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java:[57,18]
cannot find symbol
[ERROR] symbol: class TimelineClient
[ERROR] location: class org.apache.hadoop.hive.ql.hooks.ATSHook
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:[1351,9]
cannot find symbol
[ERROR] symbol: class HadoopArchives
[ERROR] location: class org.apache.hadoop.hive.ql.exec.DDLTask
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:[1351,34]
cannot find symbol
[ERROR] symbol: class HadoopArchives
[ERROR] location: class org.apache.hadoop.hive.ql.exec.DDLTask
[ERROR]
/Users/nishantkelkar/hive/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java:[72,26]
cannot find symbol
[ERROR] symbol: variable TimelineClient
[ERROR] location: class org.apache.hadoop.hive.ql.hooks.ATSHook
[ERROR] - [Help 1]
*---END---*
I looked at the pom.xml, and the following is highlighted in red:
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-archives/artifactId
version${hadoop-23.version}/version
optionaltrue/optional
/dependency
This basically means that IntelliJ/Maven was not able to find the
dependency under the Maven repo. I saw the ticket HIVE-9048
https://issues.apache.org/jira/browse/HIVE-9048 and HadoopArchives should
be a sub-dependency of the hadoop-tools dependency right?
Here's a copy of the relevant mirrors that I'm using in my settings.xml:
mirror
idConjars/id
mirrorOf*/mirrorOf
nameUS Conjars/name
urlhttp://conjars.org/repo/url
/mirror
mirror
idUS/id
mirrorOfcentral/mirrorOf
nameUS Central/name
urlhttp://repo.maven.apache.org/maven2/url
/mirror
Does anyone have an idea why this could be happening? I was kind of
expecting just cloning from Git, and running 'mvn clean compile' and seeing
everything pass and build with success.
Best Regards,
Nishant Kelkar
Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]
On 六月 23, 2015, 1:31 p.m., Xuefu Zhang wrote: ql/src/test/results/clientpositive/spark/groupby10.q.out, line 60 https://reviews.apache.org/r/34757/diff/3-4/?file=988071#file988071line60 Interesting. How come we got more stages now? Not sure, introduced by latest merge from trunk. - chengxiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88966 --- On 六月 23, 2015, 7:24 a.m., chengxiang li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/ --- (Updated 六月 23, 2015, 7:24 a.m.) Review request for hive and Xuefu Zhang. Bugs: HIVE-10844 https://issues.apache.org/jira/browse/HIVE-10844 Repository: hive-git Description --- Some Hive queries(like TPCDS Q39) may share the same subquery, which translated into sperate, but equivalent Works in SparkWork, combining these equivalent Works into a single one would help to benifit from following dynamic RDD caching optimization. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 ql/src/test/results/clientpositive/spark/groupby10.q.out dd9d9fe ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 2cb126d ql/src/test/results/clientpositive/spark/groupby8.q.out c249b61 ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 2fb1d73 ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 ql/src/test/results/clientpositive/spark/union5.q.out afee988 ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 ql/src/test/results/clientpositive/spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li
Re: Review Request 35107: HIVE-6791 Support variable substition for Beeline shell command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/ --- (Updated June 24, 2015, 3:39 p.m.) Review request for hive, chinna and Xuefu Zhang. Changes --- Update patch addressing Xuefu's comments Bugs: HIVE-6791 https://issues.apache.org/jira/browse/HIVE-6791 Repository: hive-git Description --- Summary: 1) move the beeline-cli convertor to the place where cli is executed(class **Commands**) 2) support substitution for source command 3) add some unit test for substitution 4) add one way to get the configuration from HS2 Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java b7d2f2e beeline/src/java/org/apache/hive/beeline/Commands.java a42baa3 beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java 6cbb030 cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java d62fd5c common/src/java/org/apache/hadoop/hive/conf/HiveVariableSource.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/VariableSubstitution.java PRE-CREATION common/src/test/org/apache/hadoop/hive/conf/TestVariableSubstitution.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/Driver.java 338e755 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java a5f0a7f ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java e8b1d96 ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java 0558c53 ql/src/java/org/apache/hadoop/hive/ql/processors/CompileProcessor.java 25ce168 ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java 9052c82 ql/src/java/org/apache/hadoop/hive/ql/processors/DfsProcessor.java cc0414d ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java bc9254c service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 33ee16b Diff: https://reviews.apache.org/r/35107/diff/ Testing --- Unit test passed Thanks, cheng xu
[jira] [Created] (HIVE-11090) ordering issues with windows unit test runs
Matt McCline created HIVE-11090: --- Summary: ordering issues with windows unit test runs Key: HIVE-11090 URL: https://issues.apache.org/jira/browse/HIVE-11090 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 1.2.1 Reporter: Matt McCline Assignee: Matt McCline -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]
On 六月 19, 2015, 1:47 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java, line 98 https://reviews.apache.org/r/34757/diff/3/?file=988066#file988066line98 I think the recursion should go on even if there is only one child for a given work. For examle, if we have: w1 | w2 | w3 / \ w4 w5 Even if each of w1 and w2 has only one child, it's still possible that we can combine w4 and w5. created HIVE-11082 to track this. On 六月 19, 2015, 1:47 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java, line 207 https://reviews.apache.org/r/34757/diff/3/?file=988066#file988066line207 Could you explain the reason here? add comments in latest patch. While combine multi equivalent works into single one, we need to update all the references to the replaced works. leave works output should be read by further SparkWork/FetchWork, we does not able to update work reference across SparkWork, so combine leave works may lead to error. - chengxiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88537 --- On 六月 23, 2015, 7:24 a.m., chengxiang li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/ --- (Updated 六月 23, 2015, 7:24 a.m.) Review request for hive and Xuefu Zhang. Bugs: HIVE-10844 https://issues.apache.org/jira/browse/HIVE-10844 Repository: hive-git Description --- Some Hive queries(like TPCDS Q39) may share the same subquery, which translated into sperate, but equivalent Works in SparkWork, combining these equivalent Works into a single one would help to benifit from following dynamic RDD caching optimization. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 ql/src/test/results/clientpositive/spark/groupby10.q.out dd9d9fe ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 2cb126d ql/src/test/results/clientpositive/spark/groupby8.q.out c249b61 ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 2fb1d73 ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 ql/src/test/results/clientpositive/spark/union5.q.out afee988 ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4
[jira] [Created] (HIVE-11091) Unable to load data into hive table using Load data local inapth command from unix named pipe
Manoranjan Sahoo created HIVE-11091: --- Summary: Unable to load data into hive table using Load data local inapth command from unix named pipe Key: HIVE-11091 URL: https://issues.apache.org/jira/browse/HIVE-11091 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 0.14.0 Environment: Unix,MacOS Reporter: Manoranjan Sahoo Priority: Blocker Unable to load data into hive table from unix named pipe in Hive 0.14.0 Please find below the execution details in env ( Hadoop2.6.0 + Hive 0.14.0): $ mkfifo /tmp/test.txt $ hive hive create table test(id bigint,name string); OK Time taken: 1.018 seconds hive LOAD DATA LOCAL INPATH '/tmp/test.txt' OVERWRITE INTO TABLE test; Loading data to table default.test Failed with exception addFiles: filesystem error in check phase FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask But in Hadoop 1.3 and hive 0.11.0 it works fine: hive LOAD DATA LOCAL INPATH '/tmp/test.txt' OVERWRITE INTO TABLE test; Copying data from file:/tmp/test.txt Copying file: file:/tmp/test.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] hive pull request: Update from original
GitHub user gunnartangring opened a pull request: https://github.com/apache/hive/pull/42 Update from original You can merge this pull request into a Git repository by running: $ git pull https://github.com/gunnartangring/hive trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/42.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #42 commit 474f12672485cc41b041d50c1e131c20a3175b7f Author: Jason Dere jd...@apache.org Date: 2015-02-09T23:06:17Z HIVE-9520: Create NEXT_DAY UDF (Alexander Pivovarov via Jason Dere) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1658600 13f79535-47bb-0310-9956-ffa450edef68 commit 3edf89802b95259e90d6c20fc56f5bf29426b2a6 Author: Ashutosh Chauhan hashut...@apache.org Date: 2015-02-10T00:46:21Z HIVE-9228 : Problem with subquery using windowing functions (Navis via Ashutosh Chauhan) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1658606 13f79535-47bb-0310-9956-ffa450edef68 commit efa823f4b80d2ca3005da4479cc0a67f96597644 Author: Navis Ryu na...@apache.org Date: 2015-02-10T06:47:09Z HIVE-9507 : Make LATERAL VIEW inline(expression) mytable tolerant to nulls (Navis reviewed by Ashutosh Chauhan) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1658625 13f79535-47bb-0310-9956-ffa450edef68 commit 1e39f8e23b0fda4507ece1729974a73d994aff9d Author: Brock Noland br...@apache.org Date: 2015-02-10T17:49:19Z HIVE-9454 - Test failures due to new Calcite version (Laljo via Brock) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1658766 13f79535-47bb-0310-9956-ffa450edef68 commit 3e7a258f910c78110405c3564d1ab3122281ae27 Author: Xuefu Zhang xu...@apache.org Date: 2015-02-10T19:38:40Z HIVE-9574: Lazy computing in HiveBaseFunctionResultList may hurt performance [Spark Branch] (Jimmy via Xuefu) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/spark@1658786 13f79535-47bb-0310-9956-ffa450edef68 commit 02b20e97133d8b44ebb73ac42b20bce0be19d715 Author: Xuefu Zhang xu...@apache.org Date: 2015-02-10T19:46:42Z HIVE-9627: Add cbo_gby_empty.q.out for Spark [Spark Branch] (Jimmy via Xuefu) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/spark@1658788 13f79535-47bb-0310-9956-ffa450edef68 commit c6d850dcc5bcedb6b17aff4cb804585a79ae2505 Author: Xuefu Zhang xu...@apache.org Date: 2015-02-11T05:27:29Z HIVE-9646: Beeline doesn't show Spark job progress info [Spark Branch] (Jimmy via Xuefu) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/spark@1658877 13f79535-47bb-0310-9956-ffa450edef68 commit f7248105508426c71b655c5dedeea69eec8fb4e5 Author: Jason Dere jd...@apache.org Date: 2015-02-11T10:22:41Z HIVE-9594: Add qtests for LAST_DAY udf (Alexander Pivovarov via Jason Dere) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1658914 13f79535-47bb-0310-9956-ffa450edef68 commit e1b0674e5bca7773fc8656ec95541aa95dfbad4a Author: Brock Noland br...@apache.org Date: 2015-02-11T15:48:33Z HIVE-9612 - Turn off DEBUG logging for Lazy Objects for tests (Sergio via Brock/Szehon) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1658991 13f79535-47bb-0310-9956-ffa450edef68 commit 3004101d1aac5b0e39f2777174aa981e4930e5d0 Author: Brock Noland br...@apache.org Date: 2015-02-11T18:10:09Z HIVE-9651 - FileSinkOperator does not pass in conf to serde.initialize (Brock via Szehon) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1659027 13f79535-47bb-0310-9956-ffa450edef68 commit 4df3c6becb7002a27e10374c4a000529c4f266a4 Author: Prasanth J prasan...@apache.org Date: 2015-02-11T20:46:00Z HIVE-9652: Tez in place updates should detect redirection of STDERR (Prasanth Jayachandran reviewed by Gopal V) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1659065 13f79535-47bb-0310-9956-ffa450edef68 commit 5d175fcda6c5dad410d98ce963d1e16ef0f06cd1 Author: Vaibhav Gumashta vgumas...@apache.org Date: 2015-02-11T21:29:28Z HIVE-9621: HiveServer2 http mode - embedded jetty should use SynchronousQueue (Vaibhav Gumashta reviewed by Thejas Nair) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1659098 13f79535-47bb-0310-9956-ffa450edef68 commit 09ee4803d015c51e864ee923fd60af50e972074c Author: Owen O'Malley omal...@apache.org Date: 2015-02-11T23:00:45Z HIVE-9593. ORC Reader should ignore unknown metadata streams (omalley) git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1659106 13f79535-47bb-0310-9956-ffa450edef68 commit 4bdf3acdb0126adde9a016d458fa83232412133c Author: Brock Noland br...@apache.org Date: 2015-02-12T04:52:38Z HIVE-9648 - Null check key provider before doing set (Brock via Chao) git-svn-id:
[GitHub] hive pull request: Update from original
Github user gunnartangring closed the pull request at: https://github.com/apache/hive/pull/42 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (HIVE-11093) ALTER TABLE for avro backed tables
Julian Keppel created HIVE-11093:
Summary: ALTER TABLE for avro backed tables
Key: HIVE-11093
URL: https://issues.apache.org/jira/browse/HIVE-11093
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.1.0
Environment: Cloudera CDH 5.4.0
Reporter: Julian Keppel
I use Hive Version 1.1.0 in Cloudera CDH 5.4.0.
I have created an external table:
CREATE EXTERNAL TABLE *tableA*
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS
INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 'location'
TBLPROPERTIES ('avro.schema.url'=schema_location');
Now I wanted to create a managed table with exactly the same columns except
of one additional column (a generated hash key). The only idea I had was to
create the table with: CREATE TABLE *tableB* LIKE *tableA*;
And then add the additional column with: ALTER TABLE *tableB* ADD COLUMNS
(new_column INT);
The statements run without any errors or exceptions (even in the log files
under /var/log/hive) but the new column doesn't appear.
With a CSV table it worked properly. So I think this has to do with the fact
that its a avro backed table. But according to this ticket
(https://issues.apache.org/jira/browse/HIVE-7446) the feature was added in
version 0.14.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11092) First delta of an ORC ACID table contains incorrect schema
Elliot West created HIVE-11092:
--
Summary: First delta of an ORC ACID table contains incorrect schema
Key: HIVE-11092
URL: https://issues.apache.org/jira/browse/HIVE-11092
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Elliot West
Assignee: Elliot West
Priority: Minor
I've been reading ORC ACID data that backs transactional tables from a process
external to Hive. Initially I tried to use 'schema on read' but found some
inconsistencies in the schema returned from the initial delta file and
subsequent delta and base files. To reproduce the issue by example:
{code}
CREATE TABLE base_table ( id int, message string )
PARTITIONED BY ( continent string, country string )
CLUSTERED BY (id) INTO 1 BUCKETS
STORED AS ORC
TBLPROPERTIES ('transactional' = 'true');
INSERT INTO TABLE base_table PARTITION (continent = 'Asia', country = 'India')
VALUES (1, 'x'), (2, 'y'), (3, 'z');
UPDATE base_table SET message = 'updated' WHERE id = 1;
{code}
Now examining the raw data with the {{orcfiledump}} utility (edited for
brevity):
{code}
cd hive/warehouse/base_table/continent=Asia/country=India/
hive --orcfiledump delta_001_001/bucket_0
Type:
structoperation:int,originalTransaction:bigint,bucket:int,rowId:bigint,currentTransaction:bigint,row:struct_col0:int,_col1:string
hive --orcfiledump delta_002_002/bucket_0
Type:
structoperation:int,originalTransaction:bigint,bucket:int,rowId:bigint,currentTransaction:bigint,row:structid:int,message:string
{code}
The row schema for the first delta that resulted from the inserts has its field
names erased: {{row:struct_col0:int,_col1:string}}, whereas the delta for the
update reports the correct schema: {{row:structid:int,message:string}}. I
have also checked this with my own reader code so am confident that
{{FileDump}} is not at fault.
I believe that the row field names, and hence schema, should be consistent
across all ORC files in the ACID data set. This will enable schema on read with
field access by name (not index), which is currently not possible. Therefore
I'd like to get this issue resolved.
I'm happy to work on this, however after working through {{OrcRecordUpdater}}
and {{FileSinkOperator}} and related tests I've failed to reproduce or isolate
the issue at a smaller scale. I'd be grateful for some suggestions on where to
look next.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11094) Beeline redirecting all output to ErrorStream
Jesus Camacho Rodriguez created HIVE-11094: -- Summary: Beeline redirecting all output to ErrorStream Key: HIVE-11094 URL: https://issues.apache.org/jira/browse/HIVE-11094 Project: Hive Issue Type: Bug Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Beeline is sending all output to ErrorStream, instead of using OutputStream for info or debug information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11099) Add support for running negative q-tests [Spark Branch]
Mohit Sabharwal created HIVE-11099: -- Summary: Add support for running negative q-tests [Spark Branch] Key: HIVE-11099 URL: https://issues.apache.org/jira/browse/HIVE-11099 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Add support for TestSparkNegativeCliDriver TestMiniSparkOnYarnNegativeCliDriver to negative q-tests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11098) Load aux jars without requiring restart
Andrés Cordero created HIVE-11098: - Summary: Load aux jars without requiring restart Key: HIVE-11098 URL: https://issues.apache.org/jira/browse/HIVE-11098 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.13.0 Reporter: Andrés Cordero Currently, loading auxiliary jars (e.g. for UDFs) requires a restart, even for files under the path. Replacing an auxiliary jar also requires a restart to pick up the new contents. Would be useful if changes were picked up automatically based on filesystem notifications, or provide a manual method of forcing a reload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11102) ReaderImpl: getColumnIndicesFromNames does not work for ACID tables
Gopal V created HIVE-11102:
--
Summary: ReaderImpl: getColumnIndicesFromNames does not work for
ACID tables
Key: HIVE-11102
URL: https://issues.apache.org/jira/browse/HIVE-11102
Project: Hive
Issue Type: Bug
Components: File Formats
Affects Versions: 1.3.0, 1.2.1, 2.0.0
Reporter: Gopal V
Assignee: Gopal V
ORC reader impl does not estimate the size of ACID data files correctly.
{code}
Caused by: java.lang.IndexOutOfBoundsException: Index: 0
at java.util.Collections$EmptyList.get(Collections.java:3212)
at
org.apache.hadoop.hive.ql.io.orc.OrcProto$Type.getSubtypes(OrcProto.java:12240)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.getColumnIndicesFromNames(ReaderImpl.java:651)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.getRawDataSizeOfColumns(ReaderImpl.java:634)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:938)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:847)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:713)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11101) Vectorization decimal precision issue in vectorization_short_regress.q
Matt McCline created HIVE-11101:
---
Summary: Vectorization decimal precision issue in
vectorization_short_regress.q
Key: HIVE-11101
URL: https://issues.apache.org/jira/browse/HIVE-11101
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Noticed one query result line in vectorization_short_regress.q is different
when that test is run without vectorization.
It is a decimal precision issue??
{code}
1785c1797
1969-12-31 16:00:04.063 04XP4DrTCblC788515601.0 79.553
-1452617198 15601 -407009.58195572987 -15858 -511684.9
-15601.0158740.1750002 -6432.15344526 -79.553 NULL
-15601.0-2.43391201E8
---
1969-12-31 16:00:04.063 04XP4DrTCblC788515601.0 79.553
-1452617198 15601 -407009.58195572987 -15858 -511684.9
-15601.0158740.1750002 -6432.0 -79.553 NULL-15601.0
-2.43391201E8
1886a1899
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 35107: HIVE-6791 Support variable substition for Beeline shell command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/#review89251 --- beeline/src/java/org/apache/hive/beeline/Commands.java (line 717) https://reviews.apache.org/r/35107/#comment141839 Nit: method naming. substitution() - substituteVariables() or substitue() beeline/src/java/org/apache/hive/beeline/Commands.java (line 791) https://reviews.apache.org/r/35107/#comment141854 Do we need to close statement when done? beeline/src/java/org/apache/hive/beeline/Commands.java (line 814) https://reviews.apache.org/r/35107/#comment141842 Does this mean that env and sys variables are not being substituted for shell command? beeline/src/java/org/apache/hive/beeline/Commands.java (line 861) https://reviews.apache.org/r/35107/#comment141843 Don't we need to close reader object? ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java (line 145) https://reviews.apache.org/r/35107/#comment141846 Could we keep @Override at a separate line? Same for other places. - Xuefu Zhang On June 24, 2015, 7:39 a.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/ --- (Updated June 24, 2015, 7:39 a.m.) Review request for hive, chinna and Xuefu Zhang. Bugs: HIVE-6791 https://issues.apache.org/jira/browse/HIVE-6791 Repository: hive-git Description --- Summary: 1) move the beeline-cli convertor to the place where cli is executed(class **Commands**) 2) support substitution for source command 3) add some unit test for substitution 4) add one way to get the configuration from HS2 Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java b7d2f2e beeline/src/java/org/apache/hive/beeline/Commands.java a42baa3 beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java 6cbb030 cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java d62fd5c common/src/java/org/apache/hadoop/hive/conf/HiveVariableSource.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/VariableSubstitution.java PRE-CREATION common/src/test/org/apache/hadoop/hive/conf/TestVariableSubstitution.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/Driver.java 338e755 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java a5f0a7f ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java e8b1d96 ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java 0558c53 ql/src/java/org/apache/hadoop/hive/ql/processors/CompileProcessor.java 25ce168 ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java 9052c82 ql/src/java/org/apache/hadoop/hive/ql/processors/DfsProcessor.java cc0414d ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java bc9254c service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 33ee16b Diff: https://reviews.apache.org/r/35107/diff/ Testing --- Unit test passed Thanks, cheng xu
[jira] [Created] (HIVE-11100) Beeline should escape semi-colon in queries
Chaoyu Tang created HIVE-11100: -- Summary: Beeline should escape semi-colon in queries Key: HIVE-11100 URL: https://issues.apache.org/jira/browse/HIVE-11100 Project: Hive Issue Type: Improvement Components: Beeline Affects Versions: 1.2.0 Reporter: Chaoyu Tang Assignee: Chaoyu Tang Priority: Minor Beeline should escape the semicolon in queries. for example, the query like followings: CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ';' LINES TERMINATED BY '\n'; or CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' LINES TERMINATED BY '\n'; both failed. But the 2nd query with semicolon escaped with \ works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 35846: HIVE-11099: Add support for running negative q-tests [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35846/ --- Review request for hive. Bugs: HIVE-11099 https://issues.apache.org/jira/browse/HIVE-11099 Repository: hive-git Description --- HIVE-11099: Add support for running negative q-tests [Spark Branch] Add support for TestSparkNegativeCliDriver and TestMiniSparkOnYarnNegativeCliDriver for negative q-tests Adding empty spark.query.negative.files and miniSparkOnYarn.query.negative.files lists in testconfiguration.properties. These lists can be populated as we enable negative q-tests. HIVE-11032 is a follow-up patch which will enable some of these tests Diffs - itests/qtest-spark/pom.xml dcb76230584857294e0d7346da8e7353084ea3ae itests/src/test/resources/testconfiguration.properties 7b7559a9590803f2528a24180a962013881cba1b Diff: https://reviews.apache.org/r/35846/diff/ Testing --- Tested with a negative q-test: mvn test -Dmodule=ql -Dtest=TestSparkNegativeCliDriver -Dqfile=groupby2_map_skew_multi_distinct.q -Phadoop-2 Thanks, Mohit Sabharwal
[jira] [Created] (HIVE-11095) SerDeUtils another bug ,when Text is reused
xiaowei wang created HIVE-11095: --- Summary: SerDeUtils another bug ,when Text is reused Key: HIVE-11095 URL: https://issues.apache.org/jira/browse/HIVE-11095 Project: Hive Issue Type: Bug Components: API, CLI Affects Versions: 1.2.0, 1.0.0, 0.14.0 Environment: Hadoop 2.3.0-cdh5.0.0 Hive 0.14 Reporter: xiaowei wang Assignee: xiaowei wang Priority: Critical Fix For: 1.2.0 the method transformTextFromUTF8 have a bug, When i query data from a lzo table ， I found in results ： the length of the current row is always largr than the previous row， and sometimes，the current row contains the contents of the previous row。 For example ，i execute a sql ,select * from web_searchhub where logdate=2015061003, the result of sql see blow.Notice that ,the second row content contains the first row content. INFO [03:00:05.589] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42098,session=3151,thread=254 2015061003 INFO [03:00:05.594] 18941e66-9962-44ad-81bc-3519f47ba274 session=901,thread=223ession=3151,thread=254 2015061003 The content of origin lzo file content see below ,just 2 rows. INFO [03:00:05.635] b88e0473-7530-494c-82d8-e2d2ebd2666c_forweb session=3148,thread=285 INFO [03:00:05.635] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42095,session=3148,thread=285 I think this error is caused by the Text reuse,and I found the solutions . Addicational, table create sql is : CREATE EXTERNAL TABLE `web_searchhub`( `line` string) PARTITIONED BY ( `logdate` string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' U' WITH SERDEPROPERTIES ( 'serialization.encoding'='GBK') STORED AS INPUTFORMAT com.hadoop.mapred.DeprecatedLzoTextInputFormat OUTPUTFORMAT org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat; LOCATION 'viewfs://nsX/user/hive/warehouse/raw.db/web/web_searchhub' ； -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]
On June 23, 2015, 1:31 p.m., Xuefu Zhang wrote: ql/src/test/results/clientpositive/spark/groupby10.q.out, line 60 https://reviews.apache.org/r/34757/diff/3-4/?file=988071#file988071line60 Interesting. How come we got more stages now? chengxiang li wrote: Not sure, introduced by latest merge from trunk. Okay. Got it. Thanks. - Xuefu --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88966 --- On June 23, 2015, 7:24 a.m., chengxiang li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/ --- (Updated June 23, 2015, 7:24 a.m.) Review request for hive and Xuefu Zhang. Bugs: HIVE-10844 https://issues.apache.org/jira/browse/HIVE-10844 Repository: hive-git Description --- Some Hive queries(like TPCDS Q39) may share the same subquery, which translated into sperate, but equivalent Works in SparkWork, combining these equivalent Works into a single one would help to benifit from following dynamic RDD caching optimization. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 ql/src/test/results/clientpositive/spark/groupby10.q.out dd9d9fe ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 2cb126d ql/src/test/results/clientpositive/spark/groupby8.q.out c249b61 ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 2fb1d73 ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 ql/src/test/results/clientpositive/spark/union5.q.out afee988 ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 ql/src/test/results/clientpositive/spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li
Re: Review Request 35792: HIVE-10438 - Architecture for ResultSet Compression via external plugin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35792/#review89285 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2202) https://reviews.apache.org/r/35792/#comment141873 1. Remove leading spaces or tabs. 2. The default should be false conf/hive-site.xml (line 22) https://reviews.apache.org/r/35792/#comment141875 We probably don't want to add anything here, as these properties have default values which are sufficient. service/src/java/org/apache/hive/service/cli/ColumnCompressor.java (line 15) https://reviews.apache.org/r/35792/#comment141880 Documentation for these public APIs? service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java (line 11) https://reviews.apache.org/r/35792/#comment141878 1. Why hashtable? We should use HashMap or ConcurrentHashMap rather than HashTable. 2. will break people building Hive with Java 6. service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java (line 19) https://reviews.apache.org/r/35792/#comment141876 Remove trailing spaces. Same for other occurrences. First round of review comments. Most cosmatic. More will come. - Xuefu Zhang On June 23, 2015, 8:39 p.m., Rohit Dholakia wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35792/ --- (Updated June 23, 2015, 8:39 p.m.) Review request for hive, Vaibhav Gumashta and Xuefu Zhang. Repository: hive-git Description --- This patch enables ResultSet compression for Hive using external plugins. The patch proposes a plugin architecture that enables using external plugins to compress ResultSets on-the-fly. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df conf/hive-site.xml dab494e service/if/TCLIService.thrift baf583f service/src/gen/thrift/gen-cpp/TCLIService_types.h 7bceabd service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 86eeea3 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TEnColumn.java PRE-CREATION service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java 4f157ad service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java c973fcc service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionReq.java c048161 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TOpenSessionResp.java 351f78b service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java a4279d2 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRowSet.java d16c8a4 service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TStatus.java 24a746e service/src/gen/thrift/gen-py/TCLIService/ttypes.py 068727c service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b482533 service/src/java/org/apache/hive/service/cli/Column.java 2e21f18 service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java 47a582e service/src/java/org/apache/hive/service/cli/ColumnCompressor.java PRE-CREATION service/src/java/org/apache/hive/service/cli/ColumnCompressorService.java PRE-CREATION service/src/java/org/apache/hive/service/cli/EncodedColumnBasedSet.java PRE-CREATION service/src/java/org/apache/hive/service/cli/RowSetFactory.java e8f68ea service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java dfb7faa Diff: https://reviews.apache.org/r/35792/diff/ Testing --- Testing has been done using a docker container-based query submitter that has an integer decompressor as part of it. Using the integer compressor (also provided) and the decompressor, the end-to-end functionality can be observed. File Attachments Patch file https://reviews.apache.org/media/uploaded/files/2015/06/23/16aa08f8-2393-460a-83ef-72464fc537db__HIVE-10438.patch Thanks, Rohit Dholakia
[jira] [Created] (HIVE-11103) Add banker's rounding BROUND UDF
Alexander Pivovarov created HIVE-11103:
--
Summary: Add banker's rounding BROUND UDF
Key: HIVE-11103
URL: https://issues.apache.org/jira/browse/HIVE-11103
Project: Hive
Issue Type: New Feature
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Banker's rounding: the value is rounded to the nearest even number. Also known
as Gaussian rounding, and, in German, mathematische Rundung.
Example
{code}
2 digits2 digits
UnroundedStandard roundingGaussian rounding
54.1754 54.18 54.18
343.2050 343.21 343.20
+106.2038+106.20+106.20
=======
503.5842 503.59 503.58
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11104) Select operator doesn't propagate constants appearing in expressions
Ashutosh Chauhan created HIVE-11104: --- Summary: Select operator doesn't propagate constants appearing in expressions Key: HIVE-11104 URL: https://issues.apache.org/jira/browse/HIVE-11104 Project: Hive Issue Type: Bug Components: Logical Optimizer Affects Versions: 1.2.0, 1.0.0, 0.14.0, 1.1.0, 1.2.1 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34757: HIVE-10844: Combine equivalent Works for HoS[Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review89155 --- Ship it! - Xuefu Zhang On June 23, 2015, 7:24 a.m., chengxiang li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/ --- (Updated June 23, 2015, 7:24 a.m.) Review request for hive and Xuefu Zhang. Bugs: HIVE-10844 https://issues.apache.org/jira/browse/HIVE-10844 Repository: hive-git Description --- Some Hive queries(like TPCDS Q39) may share the same subquery, which translated into sperate, but equivalent Works in SparkWork, combining these equivalent Works into a single one would help to benifit from following dynamic RDD caching optimization. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 19aae70 ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 ql/src/test/results/clientpositive/spark/groupby10.q.out dd9d9fe ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out 2cb126d ql/src/test/results/clientpositive/spark/groupby8.q.out c249b61 ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 2fb1d73 ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 ql/src/test/results/clientpositive/spark/union5.q.out afee988 ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 ql/src/test/results/clientpositive/spark/union_top_level.q.out dede1ef Diff: https://reviews.apache.org/r/34757/diff/ Testing --- Thanks, chengxiang li
Re: Review Request 35107: HIVE-6791 Support variable substition for Beeline shell command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/ --- (Updated June 25, 2015, 1:54 p.m.) Review request for hive, chinna and Xuefu Zhang. Changes --- code clean Bugs: HIVE-6791 https://issues.apache.org/jira/browse/HIVE-6791 Repository: hive-git Description --- Summary: 1) move the beeline-cli convertor to the place where cli is executed(class **Commands**) 2) support substitution for source command 3) add some unit test for substitution 4) add one way to get the configuration from HS2 Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java b7d2f2e beeline/src/java/org/apache/hive/beeline/Commands.java a42baa3 beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java 6cbb030 cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java d62fd5c common/src/java/org/apache/hadoop/hive/conf/HiveVariableSource.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/VariableSubstitution.java PRE-CREATION common/src/test/org/apache/hadoop/hive/conf/TestVariableSubstitution.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/Driver.java 338e755 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java a5f0a7f ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java e8b1d96 ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java 0558c53 ql/src/java/org/apache/hadoop/hive/ql/processors/CompileProcessor.java 25ce168 ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java 9052c82 ql/src/java/org/apache/hadoop/hive/ql/processors/DfsProcessor.java cc0414d ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java bc9254c service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 33ee16b Diff: https://reviews.apache.org/r/35107/diff/ Testing --- Unit test passed Thanks, cheng xu
Re: Review Request 35107: HIVE-6791 Support variable substition for Beeline shell command
On June 25, 2015, 6:21 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java, line 146 https://reviews.apache.org/r/35107/diff/3/?file=991115#file991115line146 Could we keep @Override at a separate line? Same for other places. I am using the code-style file(eclipse-styles.xml) under the dev-support folder. Seems annotation before class/method is not wrapped. Anyway, I just update all the places in this patch. Thank you for figuring this out. On June 25, 2015, 6:21 a.m., Xuefu Zhang wrote: beeline/src/java/org/apache/hive/beeline/Commands.java, line 820 https://reviews.apache.org/r/35107/diff/3/?file=991102#file991102line820 Does this mean that env and sys variables are not being substituted for shell command? No, this method is only used for retrieving hive configurations. For env and sys variables, they are subsituted by VariableSubstitution. - cheng --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/#review89251 --- On June 25, 2015, 1:54 p.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/ --- (Updated June 25, 2015, 1:54 p.m.) Review request for hive, chinna and Xuefu Zhang. Bugs: HIVE-6791 https://issues.apache.org/jira/browse/HIVE-6791 Repository: hive-git Description --- Summary: 1) move the beeline-cli convertor to the place where cli is executed(class **Commands**) 2) support substitution for source command 3) add some unit test for substitution 4) add one way to get the configuration from HS2 Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java b7d2f2e beeline/src/java/org/apache/hive/beeline/Commands.java a42baa3 beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java 6cbb030 cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java d62fd5c common/src/java/org/apache/hadoop/hive/conf/HiveVariableSource.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/VariableSubstitution.java PRE-CREATION common/src/test/org/apache/hadoop/hive/conf/TestVariableSubstitution.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/Driver.java 338e755 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java a5f0a7f ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java e8b1d96 ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java 0558c53 ql/src/java/org/apache/hadoop/hive/ql/processors/CompileProcessor.java 25ce168 ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java 9052c82 ql/src/java/org/apache/hadoop/hive/ql/processors/DfsProcessor.java cc0414d ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java bc9254c service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 33ee16b Diff: https://reviews.apache.org/r/35107/diff/ Testing --- Unit test passed Thanks, cheng xu
[jira] [Created] (HIVE-11096) Bump the parquet version to 1.7.0
Sergio Peña created HIVE-11096: -- Summary: Bump the parquet version to 1.7.0 Key: HIVE-11096 URL: https://issues.apache.org/jira/browse/HIVE-11096 Project: Hive Issue Type: Task Reporter: Sergio Peña Assignee: Ferdinand Xu Priority: Minor Parquet has moved officially as an Apache project since parquet 1.7.0. This new version does not have any bugfixes nor improvements from its last 1.6.0 version, but all imports were changed to be org.apache.parquet, and the pom.xml must use org.apache.parquet instead of com.twitter. This ticket should address those import and pom changes only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35803: HIVE-10895 ObjectStore does not close Query objects in some calls, causing a potential leak in some metastore db resources
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35803/ --- (Updated June 24, 2015, 3:13 p.m.) Review request for hive. Repository: hive-git Description --- HIVE-10895 ObjectStore does not close Query objects in some calls, causing a potential leak in some metastore db resources Diffs (updated) - metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 417ecc8 metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java PRE-CREATION Diff: https://reviews.apache.org/r/35803/diff/ Testing (updated) --- Testing has been done. Thanks, Aihua Xu
[jira] [Created] (HIVE-11097) HiveInputFormat uses String.startsWith to compare splitPath and PathToAliases
Wan Chang created HIVE-11097:
Summary: HiveInputFormat uses String.startsWith to compare
splitPath and PathToAliases
Key: HIVE-11097
URL: https://issues.apache.org/jira/browse/HIVE-11097
Project: Hive
Issue Type: Bug
Components: File Formats
Affects Versions: 1.2.0, 1.0.0, 0.13.1, 0.14.0, 0.13.0
Environment: Hive 0.13.1, Hive 2.0.0, hadoop 2.4.1
Reporter: Wan Chang
Priority: Critical
Say we have a sql as
{code}
create table if not exists test_orc_src (a int, b int, c int) stored as orc;
create table if not exists test_orc_src2 (a int, b int, d int) stored as orc;
insert overwrite table test_orc_src select 1,2,3 from src limit 1;
insert overwrite table test_orc_src2 select 1,2,4 from src limit 1;
set hive.auto.convert.join = false;
set hive.execution.engine=mr;
select
tb.c
from test.test_orc_src tb
join (select * from test.test_orc_src2) tm
on tb.a = tm.a
where tb.b = 2
{code}
The correct result is 3 but it produced no result.
I find that in HiveInputFormat.pushProjectionsAndFilters
{code}
match = splitPath.startsWith(key) || splitPathWithNoSchema.startsWith(key);
{code}
It uses startsWith to combine aliases with path, so tm will match two alias in
this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 34726: HIVE-10533
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34726/ --- (Updated June 24, 2015, 5:57 p.m.) Review request for hive, Ashutosh Chauhan and John Pullokkaran. Bugs: HIVE-10533 https://issues.apache.org/jira/browse/HIVE-10533 Repository: hive-git Description --- CBO (Calcite Return Path): Join to MultiJoin support for outer joins Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 199a358ba16e857f3b4bf9b0c07d1919eeb709d4 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveMultiJoin.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveInsertExchange4JoinRule.java 11c3d23ea7439b713b66a09a9911e47e847f2b18 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinToMultiJoinRule.java 532d7d3b56377946f6a9ad883d7b7dbf1325a8c7 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectMergeRule.java 8b90a15b5517a0b83c22b7a9bbbd90398c45184a ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java 3d1a3091bd0d9a99366da565f9c136aff0983bb7 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java a75d029ccf5b966347e5bf4b8c38695e89f80171 ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e821b1de5af947768dca916627911ed5d3433a7f ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 37012b49873495b60ea3f6cceadece1941b06b04 ql/src/test/queries/clientpositive/cbo_rp_join0.q PRE-CREATION ql/src/test/queries/clientpositive/cbo_rp_join1.q PRE-CREATION ql/src/test/results/clientpositive/cbo_rp_join0.q.out PRE-CREATION ql/src/test/results/clientpositive/cbo_rp_join1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/34726/diff/ Testing --- Thanks, Jesús Camacho Rodríguez
