[jira] [Assigned] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang reassigned HIVE-1363:
-
Assignee: Chaoyu Tang (was: Carl Steinbach)
I am running into this issue and pick up this JIRA to work on.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124348#comment-14124348
]
Pradeep Kamath commented on HIVE-1363:
--
This email account is inactive. Please contact another person at the company or
pe...@fb.com.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-1363:
--
Attachment: HIVE-1363.patch
The patch is requesting for the review: https://reviews.apache.org/r/25412/
Thanks in advanced.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Attachments: HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-1363:
--
Fix Version/s: 0.14.0
Affects Version/s: 0.14.0
Status: Patch Available (was: Open)
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0, 0.14.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25176: HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch]
On Sept. 5, 2014, 8:17 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line 1747 https://reviews.apache.org/r/25176/diff/4/?file=676862#file676862line1747 The if condition already checks fileSinkDesc.isLinkedFileSink(), how come in else block, we still need to do this? Isn't fileSinkDesc.isLinkedFileSink() consistent with fileSinkDesc.getLinkedFileSinkDesc() != null? If not, we may want to make them consistent. The situation is a bit tricky here. The isLinkedFileSink() is not consistent with fileSinkDesc.getLinkedFileSinkDesc() != null. In a special case, when the isLinkedFileSink()==false, the fileSinkDesc.getLinkedFileSinkDesc() != null. This happens when all the union all subqueries are map work. In this case, the generated graph is map1-union, map2-union, map3-union. Although the isLinkedFileSink() returns false, we still need to link the multiple filesinks together because they share the same destination directory. Otherwise, for each filesink, a set of merge and move work will get generated and their data overwrite each other. On Sept. 5, 2014, 8:17 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line 1749 https://reviews.apache.org/r/25176/diff/4/?file=676862#file676862line1749 Also, how come we set parent dir and dir to the same location? The same reason as the above one. In this case, the parent dir and the dir for each linkedfilesink are the same. On Sept. 5, 2014, 8:17 p.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java, line 189 https://reviews.apache.org/r/25176/diff/4/?file=676865#file676865line189 Does it make sense to put this code block in private method? I am fine to put this code block in a private method. - Na --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/#review52498 --- On Sept. 4, 2014, 5:03 p.m., Na Yang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/ --- (Updated Sept. 4, 2014, 5:03 p.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7870 https://issues.apache.org/jira/browse/HIVE-7870 Repository: hive-git Description --- HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch] The cause of this problem is during spark/tez task generation, the union file sink operator are cloned to two new filesink operator. The linkedfilesinkdesc info for those new filesink operators are missing. In addition, the two new filesink operators also need to be linked together. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 5ddc16d ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 379a39c ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 76fc290 ql/src/test/queries/clientpositive/union_remove_1.q c87b3fe ql/src/test/queries/clientpositive/union_remove_10.q 6701952 ql/src/test/queries/clientpositive/union_remove_11.q 4b2fa42 ql/src/test/queries/clientpositive/union_remove_12.q 69d0d0a ql/src/test/queries/clientpositive/union_remove_13.q 7605f0e ql/src/test/queries/clientpositive/union_remove_14.q a4fdfc8 ql/src/test/queries/clientpositive/union_remove_15.q e3c937b ql/src/test/queries/clientpositive/union_remove_16.q 537078b ql/src/test/queries/clientpositive/union_remove_17.q d70f3d3 ql/src/test/queries/clientpositive/union_remove_18.q 6352bc3 ql/src/test/queries/clientpositive/union_remove_19.q 8c45953 ql/src/test/queries/clientpositive/union_remove_2.q 83cd288 ql/src/test/queries/clientpositive/union_remove_20.q f80f7c1 ql/src/test/queries/clientpositive/union_remove_21.q 8963c25 ql/src/test/queries/clientpositive/union_remove_22.q b0c1ccd ql/src/test/queries/clientpositive/union_remove_23.q a1b989a ql/src/test/queries/clientpositive/union_remove_24.q ec561e0 ql/src/test/queries/clientpositive/union_remove_25.q 76c1ff5 ql/src/test/queries/clientpositive/union_remove_3.q 9617f73 ql/src/test/queries/clientpositive/union_remove_4.q cae323b ql/src/test/queries/clientpositive/union_remove_5.q 5df84e1 ql/src/test/queries/clientpositive/union_remove_6.q bfce26d ql/src/test/queries/clientpositive/union_remove_7.q 3a95674 ql/src/test/queries/clientpositive/union_remove_8.q a83a43e ql/src/test/queries/clientpositive/union_remove_9.q e71f6dd
[jira] [Commented] (HIVE-6550) SemanticAnalyzer.reset() doesn't clear all the state
[
https://issues.apache.org/jira/browse/HIVE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124356#comment-14124356
]
Hive QA commented on HIVE-6550:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666866/HIVE-6550.02.patch
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 6171 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_noscan_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_noscan_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_noscan_1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partscan_norcfile
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/663/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/663/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-663/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666866
SemanticAnalyzer.reset() doesn't clear all the state
Key: HIVE-6550
URL: https://issues.apache.org/jira/browse/HIVE-6550
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0
Reporter: Laljo John Pullokkaran
Assignee: Sergey Shelukhin
Attachments: HIVE-6550.01.patch, HIVE-6550.02.patch, HIVE-6550.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7870) Insert overwrite table query does not generate correct task plan [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124358#comment-14124358
]
Na Yang commented on HIVE-7870:
---
Removing those duplicated filesinks is hard because during the time that those
filesinks are added to the filesinkset, it is hard to know which filesink is
eventually used by the spark work. if we remove the wrong filesink from the
filesinkset, then we are not able to create the proper linkedfilesinks for the
target filesink. This will cause wrong result for the merge and move work when
hive.merge.sparkfiles is turned ON.
For example, in the following query, three duplicate filesink FS1, FS2, FS3
will be added to the filesinkset. (the number is according to the order they
are added to the filesinkset), FS2 and FS3 will be used for the subqueries of
the outer union. In addition, FS2 and FS3 have different directory when
hive.merge.sparkfiles=true.
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
However, in the following query, same as above query, three duplicate filesink
FS1, FS2, FS3 will be added to the filesinkset. But FS1 will be used for the
subqueries of the union. FS1, FS2 and FS3 all have the same directory when
hive.merge.sparkfiles=true.
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, 3 as values from inputTbl1
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
When the filesinks are added to the filesinkset, the final plan has not been
generated yet, so there is no way to know which filesink should not be added to
the set. After the final plan is generated, it is hard to detect the duplicate
filesinks and remove the right one either.
Therefore, duplicate filesinks are in the filesinkset. The potential problem
that duplicate filesinks cause is generating multiple merge and move works when
hive.merge.sparkfiles=true. This problem has been resolved in the patch by
linking those duplicate filesinks together and use a HashMap to make sure one
directory only gets processed once and only one merge and move work will be
generated for each directory no matter how many duplicate filesinks exist.
Insert overwrite table query does not generate correct task plan [Spark
Branch]
---
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Na Yang
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch, HIVE-7870.2-spark.patch,
HIVE-7870.3-spark.patch, HIVE-7870.4-spark.patch, HIVE-7870.5-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8008) NPE while reading null decimal value
[
https://issues.apache.org/jira/browse/HIVE-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124370#comment-14124370
]
Lars Francke commented on HIVE-8008:
+1
NPE while reading null decimal value
Key: HIVE-8008
URL: https://issues.apache.org/jira/browse/HIVE-8008
Project: Hive
Issue Type: Bug
Reporter: Chao
Assignee: Chao
Attachments: HIVE-8008.2.patch, HIVE-8008.3.patch, HIVE-8008.4.patch,
HIVE-8008.patch
Say you have this table {{dec_test}}:
{code}
dec decimal(10,0)
{code}
If the table has a row that is 99.5, and if we do
{code}
select * from dec_test;
{code}
it will crash with NPE:
{code}
2014-09-05 14:08:56,023 ERROR [main]: CliDriver
(SessionState.java:printError(545)) - Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:151)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1531)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:285)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:544)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:536)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:137)
... 12 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:265)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:71)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:70)
at
org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:39)
at
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:87)
... 19 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7508) Kerberos support for streaming
[ https://issues.apache.org/jira/browse/HIVE-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124383#comment-14124383 ] Lefty Leverenz commented on HIVE-7508: -- Sorry [~roshan_naik], I can't grant write permission on the wiki but Ashutosh can. Just so you know (and so everyone else knows), the usual procedure is to send the request to u...@hive.apache.org as explained in https://cwiki.apache.org/confluence/display/Hive/AboutThisWiki. Do you have a Confluence account? If not: https://cwiki.apache.org/confluence/signup.action. Kerberos support for streaming -- Key: HIVE-7508 URL: https://issues.apache.org/jira/browse/HIVE-7508 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.13.1 Reporter: Roshan Naik Assignee: Roshan Naik Labels: Streaming, TODOC14 Fix For: 0.14.0 Attachments: HIVE-7508.patch Add kerberos support for streaming to secure Hive cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7989) Optimize Windowing function performance for row frames
[
https://issues.apache.org/jira/browse/HIVE-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124393#comment-14124393
]
Hive QA commented on HIVE-7989:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666864/HIVE-7989.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6171 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/664/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/664/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-664/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666864
Optimize Windowing function performance for row frames
--
Key: HIVE-7989
URL: https://issues.apache.org/jira/browse/HIVE-7989
Project: Hive
Issue Type: Improvement
Components: PTF-Windowing
Affects Versions: 0.13.0
Reporter: Ankit Kamboj
Attachments: HIVE-7989.patch
To find aggregate value for each row, current windowing function
implementation creates a new aggregation buffer for each row, iterates over
all the rows in respective window frame, puts them in buffer and then finds
the aggregated value. This causes bottleneck for partitions with huge number
of rows because this process runs in n-square complexity (n being rows in a
partition) for each partition. So, if there are multiple partitions in a
dataset, each with millions of rows, aggregation for all rows will take days
to finish.
There is scope of optimization for row frames, for following cases:
a) For UNBOUNDED PRECEDING start and bounded end: Instead of iterating on
window frame again for each row, we can slide the end one row at a time and
aggregate, since we know the start is fixed for each row. This will have
running time linear to the size of partition.
b) For bounded start and UNBOUNDED FOLLOWING end: Instead of iterating on
window frame again for each row, we can slide the start one row at a time and
aggregate in reverse, since we know the end is fixed for each row. This will
have running time linear to the size of partition.
Also, In general for both row and value frames, we don't need to iterate over
the range and re-create aggregation buffer if the start as well as end remain
same. Instead, can re-use the previously created aggregation buffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7405) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
[ https://issues.apache.org/jira/browse/HIVE-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-7405: --- Status: Patch Available (was: In Progress) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic) -- Key: HIVE-7405 URL: https://issues.apache.org/jira/browse/HIVE-7405 Project: Hive Issue Type: Sub-task Components: Vectorization Reporter: Matt McCline Assignee: Matt McCline Attachments: HIVE-7405.1.patch, HIVE-7405.2.patch, HIVE-7405.3.patch, HIVE-7405.4.patch, HIVE-7405.5.patch, HIVE-7405.6.patch, HIVE-7405.7.patch, HIVE-7405.8.patch, HIVE-7405.9.patch, HIVE-7405.91.patch, HIVE-7405.92.patch, HIVE-7405.93.patch, HIVE-7405.94.patch, HIVE-7405.95.patch, HIVE-7405.96.patch, HIVE-7405.97.patch, HIVE-7405.98.patch, HIVE-7405.99.patch, HIVE-7405.991.patch, HIVE-7405.994.patch Vectorize the basic case that does not have any count distinct aggregation. Add a 4th processing mode in VectorGroupByOperator for reduce where each input VectorizedRowBatch has only values for one key at a time. Thus, the values in the batch can be aggregated quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8009) show compactions fails when a table with no partitions is being compacted
[
https://issues.apache.org/jira/browse/HIVE-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124396#comment-14124396
]
Damien Carol commented on HIVE-8009:
I already filed a JIRA with a patch that fix this bug HIVE-7470
show compactions fails when a table with no partitions is being compacted
-
Key: HIVE-8009
URL: https://issues.apache.org/jira/browse/HIVE-8009
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 0.13.1
Reporter: Alan Gates
Assignee: Alan Gates
When a table with no partitions is being compacted and the user does show
compactions it fails. The message from the logs is:
{quote}
org.apache.thrift.protocol.TProtocolException: Required field 'partitionname'
is unset! Struct:ShowCompactResponseElement(dbname:default, tablename:orctab,
partitionname:null, type:MAJOR, state:initiated, workerid:null, start:0,
runAs:null)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-8009) show compactions fails when a table with no partitions is being compacted
[
https://issues.apache.org/jira/browse/HIVE-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol resolved HIVE-8009.
Resolution: Duplicate
show compactions fails when a table with no partitions is being compacted
-
Key: HIVE-8009
URL: https://issues.apache.org/jira/browse/HIVE-8009
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 0.13.1
Reporter: Alan Gates
Assignee: Alan Gates
When a table with no partitions is being compacted and the user does show
compactions it fails. The message from the logs is:
{quote}
org.apache.thrift.protocol.TProtocolException: Required field 'partitionname'
is unset! Struct:ShowCompactResponseElement(dbname:default, tablename:orctab,
partitionname:null, type:MAJOR, state:initiated, workerid:null, start:0,
runAs:null)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8009) show compactions fails when a table with no partitions is being compacted
[
https://issues.apache.org/jira/browse/HIVE-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124397#comment-14124397
]
Damien Carol commented on HIVE-8009:
[~alangates] see my descritpion of the bug. I can rebase the patch.
show compactions fails when a table with no partitions is being compacted
-
Key: HIVE-8009
URL: https://issues.apache.org/jira/browse/HIVE-8009
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 0.13.1
Reporter: Alan Gates
Assignee: Alan Gates
When a table with no partitions is being compacted and the user does show
compactions it fails. The message from the logs is:
{quote}
org.apache.thrift.protocol.TProtocolException: Required field 'partitionname'
is unset! Struct:ShowCompactResponseElement(dbname:default, tablename:orctab,
partitionname:null, type:MAJOR, state:initiated, workerid:null, start:0,
runAs:null)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7991) Incorrect calculation of number of rows in JoinStatsRule.process results in overflow
[
https://issues.apache.org/jira/browse/HIVE-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124407#comment-14124407
]
Hive QA commented on HIVE-7991:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666876/HIVE-7991.1.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 6171 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union20
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/665/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/665/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-665/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666876
Incorrect calculation of number of rows in JoinStatsRule.process results in
overflow
Key: HIVE-7991
URL: https://issues.apache.org/jira/browse/HIVE-7991
Project: Hive
Issue Type: Sub-task
Components: Statistics
Affects Versions: 0.13.1
Reporter: Mostafa Mokhtar
Assignee: Prasanth J
Priority: Minor
Attachments: HIVE-7991.1.patch
This loop results in adding the parent twice incase of a 3 way join of
store_sales x date_dim x store
{code}
for (int pos = 0; pos parents.size(); pos++) {
ReduceSinkOperator parent = (ReduceSinkOperator)
jop.getParentOperators().get(pos);
Statistics parentStats = parent.getStatistics();
ListExprNodeDesc keyExprs = parent.getConf().getKeyCols();
// Parent RS may have column statistics from multiple parents.
// Populate table alias to row count map, this will be used later
to
// scale down/up column statistics based on new row count
// NOTE: JOIN with UNION as parent of RS will not have table alias
// propagated properly. UNION operator does not propagate the
table
// alias of subqueries properly to expression nodes. Hence
union20.q
// will have wrong number of rows.
SetString tableAliases =
StatsUtils.getAllTableAlias(parent.getColumnExprMap());
for (String tabAlias : tableAliases) {
rowCountParents.put(tabAlias, parentStats.getNumRows());
}
{code}
In the first join we have rowCountParents with {store_sales=120464862,
date_dim=36524} which is correct.
For the second join result rowCountParents ends up with {store=212,
store_sales=120464862, date_dim=120464862} where it should be {store=212,
store_sales=120464862, date_dim=36524}.
The result of this is that computeNewRowCount ends up multiplying row count
of store_sales x store_sales which makes the number of rows really high and
eventually over flow.
Plan snippet :
{code}
Map 1
Map Operator Tree:
TableScan
alias: store_sales
filterExpr: (((ss_sold_date_sk is not null and ss_store_sk
is not null) and ss_item_sk is not null) and ss_sold_date BETWEEN
'1999-06-01' AND '2000-05-31') (type: boolean)
Statistics: Num rows: 110339135 Data size: 4817453454 Basic
stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((ss_sold_date_sk is not null and ss_store_sk
is not null) and ss_item_sk is not null) (type: boolean)
Statistics: Num rows: 107740258 Data size: 2124353556
Basic stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {ss_sold_date_sk} {ss_item_sk} {ss_store_sk}
{ss_quantity} {ss_sales_price} {ss_sold_date}
1 {d_date_sk} {d_month_seq} {d_year} {d_moy} {d_qoy}
keys:
0 ss_sold_date_sk (type: int)
1 d_date_sk (type: int)
outputColumnNames: _col0, _col2, _col7, _col10, _col13,
[jira] [Updated] (HIVE-2390) Expand support for union types
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated HIVE-2390:
---
Attachment: HIVE-2390.1.patch
Expand support for union types
--
Key: HIVE-2390
URL: https://issues.apache.org/jira/browse/HIVE-2390
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Jakob Homan
Assignee: Suma Shivaprasad
Labels: uniontype
Fix For: 0.14.0
Attachments: HIVE-2390.1.patch, HIVE-2390.patch
When the union type was introduced, full support for it wasn't provided. For
instance, when working with a union that gets passed to LazyBinarySerde:
{noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7946) CBO: Merge CBO changes to Trunk
[
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124452#comment-14124452
]
Hive QA commented on HIVE-7946:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666892/HIVE-7946.2.patch
{color:red}ERROR:{color} -1 due to 476 failed/errored test(s), 5543 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver_accumulo_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_admin_almighty2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_view_sqlstd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_partitioned_native
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_constant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast_to_int
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_cast
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_udf1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_compile_processor
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_struct_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_union_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
[jira] [Updated] (HIVE-7946) CBO: Merge CBO changes to Trunk
[ https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7946: - Status: Open (was: Patch Available) CBO: Merge CBO changes to Trunk --- Key: HIVE-7946 URL: https://issues.apache.org/jira/browse/HIVE-7946 Project: Hive Issue Type: Bug Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-7946.1.patch, HIVE-7946.2.patch, HIVE-7946.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7946) CBO: Merge CBO changes to Trunk
[ https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7946: - Attachment: HIVE-7946.3.patch CBO: Merge CBO changes to Trunk --- Key: HIVE-7946 URL: https://issues.apache.org/jira/browse/HIVE-7946 Project: Hive Issue Type: Bug Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-7946.1.patch, HIVE-7946.2.patch, HIVE-7946.3.patch, HIVE-7946.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7946) CBO: Merge CBO changes to Trunk
[ https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7946: - Status: Patch Available (was: Open) CBO: Merge CBO changes to Trunk --- Key: HIVE-7946 URL: https://issues.apache.org/jira/browse/HIVE-7946 Project: Hive Issue Type: Bug Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-7946.1.patch, HIVE-7946.2.patch, HIVE-7946.3.patch, HIVE-7946.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7704) Create tez task for fast file merging
[
https://issues.apache.org/jira/browse/HIVE-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124502#comment-14124502
]
Hive QA commented on HIVE-7704:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666973/HIVE-7704.7.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6179 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge1
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/667/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/667/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-667/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666973
Create tez task for fast file merging
-
Key: HIVE-7704
URL: https://issues.apache.org/jira/browse/HIVE-7704
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Attachments: HIVE-7704.1.patch, HIVE-7704.2.patch, HIVE-7704.3.patch,
HIVE-7704.4.patch, HIVE-7704.4.patch, HIVE-7704.5.patch, HIVE-7704.6.patch,
HIVE-7704.7.patch
Currently tez falls back to MR task for merge file task. It will beneficial
to convert the merge file tasks to tez task to make use of the performance
gains from tez.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7470) Wrong Thrift declaration for {{ShowCompactResponseElement}}
[
https://issues.apache.org/jira/browse/HIVE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124503#comment-14124503
]
Alan Gates commented on HIVE-7470:
--
Patch looks good. Please rebase and I'll get it committed. Sorry for missing
this earlier.
Wrong Thrift declaration for {{ShowCompactResponseElement}}
---
Key: HIVE-7470
URL: https://issues.apache.org/jira/browse/HIVE-7470
Project: Hive
Issue Type: Bug
Components: Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, thrift
Fix For: 0.14.0
Attachments: HIVE-7470.1.patch
Prerequiste :
1. Remote metastore
2. Activate ACID and compactions
3. Launch ALTER TABLE foo COMPACT 'bar'
4. Call {{show_compact()}} on remote metastore
This use case throws exception in Thrift stack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8009) show compactions fails when a table with no partitions is being compacted
[
https://issues.apache.org/jira/browse/HIVE-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124504#comment-14124504
]
Alan Gates commented on HIVE-8009:
--
[~damien.carol] thanks for pointing this out. Please do rebase that one and
I'll review and commit.
show compactions fails when a table with no partitions is being compacted
-
Key: HIVE-8009
URL: https://issues.apache.org/jira/browse/HIVE-8009
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 0.13.1
Reporter: Alan Gates
Assignee: Alan Gates
When a table with no partitions is being compacted and the user does show
compactions it fails. The message from the logs is:
{quote}
org.apache.thrift.protocol.TProtocolException: Required field 'partitionname'
is unset! Struct:ShowCompactResponseElement(dbname:default, tablename:orctab,
partitionname:null, type:MAJOR, state:initiated, workerid:null, start:0,
runAs:null)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7470) Wrong Thrift declaration for {{ShowCompactResponseElement}}
[
https://issues.apache.org/jira/browse/HIVE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-7470:
-
Status: Open (was: Patch Available)
Patch looks good but needs rebased.
Wrong Thrift declaration for {{ShowCompactResponseElement}}
---
Key: HIVE-7470
URL: https://issues.apache.org/jira/browse/HIVE-7470
Project: Hive
Issue Type: Bug
Components: Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, thrift
Fix For: 0.14.0
Attachments: HIVE-7470.1.patch
Prerequiste :
1. Remote metastore
2. Activate ACID and compactions
3. Launch ALTER TABLE foo COMPACT 'bar'
4. Call {{show_compact()}} on remote metastore
This use case throws exception in Thrift stack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-1363:
--
Comment: was deleted
(was: This email account is inactive. Please contact another person at the
company or pe...@fb.com.
)
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0, 0.14.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25412: HIVE-1363: SHOW TABLE EXTENDED LIKE command does not strip single/double quotes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25412/#review52540 --- ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java https://reviews.apache.org/r/25412/#comment91339 Minor nit: could we have a local var for tableOrColumnNode.getToken().getType(), as this is long and used twice in the same method? - Xuefu Zhang On Sept. 6, 2014, 5:57 a.m., Chaoyu Tang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25412/ --- (Updated Sept. 6, 2014, 5:57 a.m.) Review request for hive. Repository: hive-git Description --- 'SHOW TABLE EXTENDED [IN|FROM database_name] LIKE identifier_with_wildcards' command does not strip single/double quotes around identifier_with_wildcards and returns empty results. It is because the identifier_with_wildcards in this query is a token type of HiveParser.StringLiteral and the method unescapeIdentifier used in BaseSemanticAnalyzer.getUnescapedName could not strip its quotes. The quotes become parts of the tablename regexp and therefore no matched tables are returned for this pattern. For the token of HiveParser.StringLiteral from this kind of query, we can use unescapeSQLString to strip their quotes. The change in this patch is to use unescapeSQLString to remove the quotes around identifier_with_wildcards. Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java b5b2b609b2bda7180219ae95fbbe42ad07ed0d97 ql/src/test/queries/clientpositive/show_tables.q f33325dbb9e410a828b3bb8b1e5bdce234025d3b ql/src/test/results/clientpositive/show_tables.q.out 0d4aa5713e22bec90c8dd68ab08de4f8ff9d6eb8 Diff: https://reviews.apache.org/r/25412/diff/ Testing --- 1. Manual test: with the changes, all following cases are working just like the other command SHOW TABLES... SHOW TABLE EXTENDED IN test_db LIKE 'ba*'; SHOW TABLE EXTENDED IN test_db LIKE ba*; SHOW TABLE EXTENDED IN test_db LIKE `ba*`; 2. six related unit tests are added to show_tables.q and they work as expected 3. submit this patch for pre-commit test to see if the changes will cause other regression. Thanks, Chaoyu Tang
[jira] [Commented] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124508#comment-14124508
]
Xuefu Zhang commented on HIVE-1363:
---
Thanks for working on this, [~ctang.cloudera]. Patch looks good to me. Minor
comment on RB.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0, 0.14.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7863) Potential null reference in TxnDbUtil#prepareDb()
[
https://issues.apache.org/jira/browse/HIVE-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124511#comment-14124511
]
Alan Gates commented on HIVE-7863:
--
+1. The HS2 unit test failure is unrelated.
Potential null reference in TxnDbUtil#prepareDb()
-
Key: HIVE-7863
URL: https://issues.apache.org/jira/browse/HIVE-7863
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-7863.1.patch
Here is related code:
{code}
Connection conn = null;
boolean committed = false;
try {
conn = getConnection();
...
} finally {
if (!committed) conn.rollback();
conn.close();
}
{code}
If getConnection() throws exception, conn.rollback() would be called -
leading to NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7078) Need file sink operators that work with ACID
[ https://issues.apache.org/jira/browse/HIVE-7078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7078: - Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Neither of the failed tests are related to this patch. Patch 2 checked in. Thank you Prasanth for the review. Need file sink operators that work with ACID Key: HIVE-7078 URL: https://issues.apache.org/jira/browse/HIVE-7078 Project: Hive Issue Type: Sub-task Components: Query Processor, Transactions Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.14.0 Attachments: HIVE-7078.2.patch, HIVE-7078.WIP.patch, HIVE-7078.patch When writing records to tables that use an OutputFormat that implements AcidOutputFormat, records will need to be written via a RecordUpdater instead of RecordWriter. Rather than add this functionality to FileSinkOperator and VectorizedFileSinkOperator I propose to add a new AcidFileSinkOperator that extends FileSinkOperator and a new AcidVectorizedFileSinkOperator that extends VectorizedFileSinkOperator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7863) Potential null reference in TxnDbUtil#prepareDb()
[
https://issues.apache.org/jira/browse/HIVE-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124538#comment-14124538
]
Lars Francke commented on HIVE-7863:
Alan, thanks for checking. I hadn't gotten around to look at the test results
in detail.
Potential null reference in TxnDbUtil#prepareDb()
-
Key: HIVE-7863
URL: https://issues.apache.org/jira/browse/HIVE-7863
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-7863.1.patch
Here is related code:
{code}
Connection conn = null;
boolean committed = false;
try {
conn = getConnection();
...
} finally {
if (!committed) conn.rollback();
conn.close();
}
{code}
If getConnection() throws exception, conn.rollback() would be called -
leading to NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8008) NPE while reading null decimal value
[
https://issues.apache.org/jira/browse/HIVE-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124539#comment-14124539
]
Hive QA commented on HIVE-8008:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666957/HIVE-8008.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6171 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/668/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/668/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-668/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666957
NPE while reading null decimal value
Key: HIVE-8008
URL: https://issues.apache.org/jira/browse/HIVE-8008
Project: Hive
Issue Type: Bug
Reporter: Chao
Assignee: Chao
Attachments: HIVE-8008.2.patch, HIVE-8008.3.patch, HIVE-8008.4.patch,
HIVE-8008.patch
Say you have this table {{dec_test}}:
{code}
dec decimal(10,0)
{code}
If the table has a row that is 99.5, and if we do
{code}
select * from dec_test;
{code}
it will crash with NPE:
{code}
2014-09-05 14:08:56,023 ERROR [main]: CliDriver
(SessionState.java:printError(545)) - Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:151)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1531)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:285)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:544)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:536)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:137)
... 12 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:265)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:71)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:70)
at
org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:39)
at
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:87)
... 19 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7984) AccumuloOutputFormat Configuration items from StorageHandler not re-set in Configuration in Tez
[
https://issues.apache.org/jira/browse/HIVE-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HIVE-7984:
-
Summary: AccumuloOutputFormat Configuration items from StorageHandler not
re-set in Configuration in Tez (was: Configuration items from StorageHandler
not passed to Tez Configuration)
AccumuloOutputFormat Configuration items from StorageHandler not re-set in
Configuration in Tez
---
Key: HIVE-7984
URL: https://issues.apache.org/jira/browse/HIVE-7984
Project: Hive
Issue Type: Bug
Components: StorageHandler, Tez
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 0.14.0
Attachments: HIVE-7984-1.diff
Ran AccumuloStorageHandler queries with Tez and found that configuration
elements that are pulled from the {{-hiveconf}} and passed to the
inputJobProperties or outputJobProperties by the AccumuloStorageHandler
aren't available inside of the Tez container.
I'm guessing that there is a disconnect from the configuration that the
StorageHandler creates and what the Tez container sees.
The HBaseStorageHandler likely doesn't run into this because it expects to
have hbase-site.xml available via tmpjars (and can extrapolate connection
information from that file). Accumulo's site configuration file is not meant
to be shared with consumers which means that this exact approach is not
sufficient.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8014) hive.in.test not set in MiniTezCli tests
Alan Gates created HIVE-8014: Summary: hive.in.test not set in MiniTezCli tests Key: HIVE-8014 URL: https://issues.apache.org/jira/browse/HIVE-8014 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.13.1 Reporter: Alan Gates Assignee: Gunther Hagleitner When the TestCli driver is run, the configuration value hive.in.test is set to true. However, when TestMiniTezCli driver is run, this value is not set, which results in some classes not realizing they are running in a test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7788) Generate plans for insert, update, and delete
[ https://issues.apache.org/jira/browse/HIVE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7788: - Status: Patch Available (was: Open) Generate plans for insert, update, and delete - Key: HIVE-7788 URL: https://issues.apache.org/jira/browse/HIVE-7788 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7788.WIP.patch, HIVE-7788.patch Insert plans needs to be generated differently for ACID tables, plus we need to be able to generate plans in the semantic analyzer for update and delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7788) Generate plans for insert, update, and delete
[ https://issues.apache.org/jira/browse/HIVE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7788: - Attachment: HIVE-7788.patch This patch adds plan generation as well as making modifications to some of the exec operators to make insert/value, update, and delete work. The patch is large, but about 2/3 of that are tests. Generate plans for insert, update, and delete - Key: HIVE-7788 URL: https://issues.apache.org/jira/browse/HIVE-7788 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7788.WIP.patch, HIVE-7788.patch Insert plans needs to be generated differently for ACID tables, plus we need to be able to generate plans in the semantic analyzer for update and delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 25414: HIVE-7788 Generate plans for insert, update, and delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/ --- Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and Thejas Nair. Bugs: HIVE-7788 https://issues.apache.org/jira/browse/HIVE-7788 Repository: hive-git Description --- This patch adds plan generation as well as making modifications to some of the exec operators to make insert/value, update, and delete work. The patch is large, but about 2/3 of that are tests. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 data/conf/tez/hive-site.xml 0b3877c itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 1a84024 itests/src/test/resources/testconfiguration.properties 99049ca metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java f1697bb ql/src/java/org/apache/hadoop/hive/ql/Context.java 7fcbe3c ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9953919 ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 4246d68 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java f018ca0 ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java e3bc3b1 ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 7f1d71b ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java b1c4441 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 913d3ac ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 264052f ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 8354ad9 ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java 32d2f7a ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2b1a345 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java 96a5d78 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java 5c711cf ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java b5b2b60 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e4a30a2 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 026efe8 ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 2dbf1c8 ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 6dce30c ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 5695f35 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c409ef5 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 789c780 ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 63ecb8d ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/test/queries/clientnegative/delete_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_partition_col.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_no_match.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_whole_partition.q PRE-CREATION ql/src/test/queries/clientpositive/insert_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_update_delete.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_dynamic_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_after_multiple_inserts.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_types.q PRE-CREATION ql/src/test/queries/clientpositive/update_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_two_cols.q PRE-CREATION ql/src/test/queries/clientpositive/update_where_no_match.q
Re: Review Request 25414: HIVE-7788 Generate plans for insert, update, and delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/ --- (Updated Sept. 6, 2014, 4:32 p.m.) Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and Thejas Nair. Bugs: HIVE-7788 https://issues.apache.org/jira/browse/HIVE-7788 Repository: hive-git Description --- This patch adds plan generation as well as making modifications to some of the exec operators to make insert/value, update, and delete work. The patch is large, but about 2/3 of that are tests. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 data/conf/tez/hive-site.xml 0b3877c itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 1a84024 itests/src/test/resources/testconfiguration.properties 99049ca metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java f1697bb ql/src/java/org/apache/hadoop/hive/ql/Context.java 7fcbe3c ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9953919 ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 4246d68 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java f018ca0 ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java e3bc3b1 ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 7f1d71b ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java b1c4441 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 913d3ac ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 264052f ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 8354ad9 ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java 32d2f7a ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2b1a345 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java 96a5d78 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java 5c711cf ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java b5b2b60 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e4a30a2 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 026efe8 ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 2dbf1c8 ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 6dce30c ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 5695f35 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c409ef5 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 789c780 ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 63ecb8d ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/test/queries/clientnegative/delete_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_partition_col.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_no_match.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_whole_partition.q PRE-CREATION ql/src/test/queries/clientpositive/insert_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_update_delete.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_dynamic_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_after_multiple_inserts.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_types.q PRE-CREATION ql/src/test/queries/clientpositive/update_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_two_cols.q PRE-CREATION
Re: Review Request 25414: HIVE-7788 Generate plans for insert, update, and delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/ --- (Updated Sept. 6, 2014, 4:32 p.m.) Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and Thejas Nair. Bugs: HIVE-7788 https://issues.apache.org/jira/browse/HIVE-7788 Repository: hive-git Description --- This patch adds plan generation as well as making modifications to some of the exec operators to make insert/value, update, and delete work. The patch is large, but about 2/3 of that are tests. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 data/conf/tez/hive-site.xml 0b3877c itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 1a84024 itests/src/test/resources/testconfiguration.properties 99049ca metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java f1697bb ql/src/java/org/apache/hadoop/hive/ql/Context.java 7fcbe3c ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9953919 ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 4246d68 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java f018ca0 ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java e3bc3b1 ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 7f1d71b ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java b1c4441 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 913d3ac ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 264052f ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 8354ad9 ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java 32d2f7a ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2b1a345 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java 96a5d78 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java 5c711cf ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java b5b2b60 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e4a30a2 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 026efe8 ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 2dbf1c8 ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 6dce30c ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 5695f35 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c409ef5 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 789c780 ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 63ecb8d ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/test/queries/clientnegative/delete_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_partition_col.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_no_match.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_whole_partition.q PRE-CREATION ql/src/test/queries/clientpositive/insert_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_update_delete.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_dynamic_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_after_multiple_inserts.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_types.q PRE-CREATION ql/src/test/queries/clientpositive/update_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_two_cols.q PRE-CREATION
[jira] [Commented] (HIVE-7788) Generate plans for insert, update, and delete
[ https://issues.apache.org/jira/browse/HIVE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124546#comment-14124546 ] Alan Gates commented on HIVE-7788: -- Posted review https://reviews.apache.org/r/25414/ Generate plans for insert, update, and delete - Key: HIVE-7788 URL: https://issues.apache.org/jira/browse/HIVE-7788 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7788.WIP.patch, HIVE-7788.patch Insert plans needs to be generated differently for ACID tables, plus we need to be able to generate plans in the semantic analyzer for update and delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7975) HS2 memory optimization: Internalizing instance fields of Thrift-generated metastore API classes
[ https://issues.apache.org/jira/browse/HIVE-7975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-7975: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thank you [~wilbur.yang]!! I have committed your patch to trunk!! HS2 memory optimization: Internalizing instance fields of Thrift-generated metastore API classes Key: HIVE-7975 URL: https://issues.apache.org/jira/browse/HIVE-7975 Project: Hive Issue Type: Improvement Reporter: Wilbur Yang Assignee: Wilbur Yang Fix For: 0.14.0 Attachments: HIVE-7975-doc.pdf, HIVE-7975.1.patch, HIVE-7975.2.patch, HIVE-7975.3.patch We should internalize the String-based instance fields of the metastore API classes FieldSchema, Partition, SerDeInfo, and StorageDescriptor in order to save memory. In a test environment with data consisting of about 1800 partitions, the proposed changes are able to save about 24% of old generation memory during a complex query. See details in the attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7975) HS2 memory optimization: Internalizing instance fields of Thrift-generated metastore API classes
[ https://issues.apache.org/jira/browse/HIVE-7975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-7975: --- Attachment: HIVE-7975.3.patch PS I made a minor change on commit and moved thrift-replacements.txt to src/main/resources and added an apache license to thrift-replacements.txt. HS2 memory optimization: Internalizing instance fields of Thrift-generated metastore API classes Key: HIVE-7975 URL: https://issues.apache.org/jira/browse/HIVE-7975 Project: Hive Issue Type: Improvement Reporter: Wilbur Yang Assignee: Wilbur Yang Fix For: 0.14.0 Attachments: HIVE-7975-doc.pdf, HIVE-7975.1.patch, HIVE-7975.2.patch, HIVE-7975.3.patch We should internalize the String-based instance fields of the metastore API classes FieldSchema, Partition, SerDeInfo, and StorageDescriptor in order to save memory. In a test environment with data consisting of about 1800 partitions, the proposed changes are able to save about 24% of old generation memory during a complex query. See details in the attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-7946) CBO: Merge CBO changes to Trunk
[
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124559#comment-14124559
]
Brock Noland edited comment on HIVE-7946 at 9/6/14 5:09 PM:
Thank you guys for doing this work! Two questions:
{noformat}
+ @Deprecated
+ public static void todo(String s) {
+ }
{noformat}
huh?
Also I see a ton of variables prefixed with m_. I understand that Hive doesn't
have a consistent coding style but I think it's fair to say that m_ is not
common in the codebase. Let's try and not do that going forward.
was (Author: brocknoland):
Thank you guys for doing this work! Two questions:
{noformat}
+ @Deprecated
+ public static void todo(String s) {
+ }
{noformat}
huh?
Also I see a ton of variables prefixed with m_. I understand there is some of
that Hive doesn't have a consistent coding style but I think it's fair to say
that m_ is not common in the codebase. Let's try and not do that going forward.
CBO: Merge CBO changes to Trunk
---
Key: HIVE-7946
URL: https://issues.apache.org/jira/browse/HIVE-7946
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Attachments: HIVE-7946.1.patch, HIVE-7946.2.patch, HIVE-7946.3.patch,
HIVE-7946.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7946) CBO: Merge CBO changes to Trunk
[
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124559#comment-14124559
]
Brock Noland commented on HIVE-7946:
Thank you guys for doing this work! Two questions:
{noformat}
+ @Deprecated
+ public static void todo(String s) {
+ }
{noformat}
huh?
Also I see a ton of variables prefixed with m_. I understand there is some of
that Hive doesn't have a consistent coding style but I think it's fair to say
that m_ is not common in the codebase. Let's try and not do that going forward.
CBO: Merge CBO changes to Trunk
---
Key: HIVE-7946
URL: https://issues.apache.org/jira/browse/HIVE-7946
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Attachments: HIVE-7946.1.patch, HIVE-7946.2.patch, HIVE-7946.3.patch,
HIVE-7946.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25412: HIVE-1363: SHOW TABLE EXTENDED LIKE command does not strip single/double quotes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25412/ --- (Updated Sept. 6, 2014, 5:15 p.m.) Review request for hive. Changes --- Change based on XueFu's comments. Thanks Xuefu for the review. Repository: hive-git Description --- 'SHOW TABLE EXTENDED [IN|FROM database_name] LIKE identifier_with_wildcards' command does not strip single/double quotes around identifier_with_wildcards and returns empty results. It is because the identifier_with_wildcards in this query is a token type of HiveParser.StringLiteral and the method unescapeIdentifier used in BaseSemanticAnalyzer.getUnescapedName could not strip its quotes. The quotes become parts of the tablename regexp and therefore no matched tables are returned for this pattern. For the token of HiveParser.StringLiteral from this kind of query, we can use unescapeSQLString to strip their quotes. The change in this patch is to use unescapeSQLString to remove the quotes around identifier_with_wildcards. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java b5b2b609b2bda7180219ae95fbbe42ad07ed0d97 ql/src/test/queries/clientpositive/show_tables.q f33325dbb9e410a828b3bb8b1e5bdce234025d3b ql/src/test/results/clientpositive/show_tables.q.out 0d4aa5713e22bec90c8dd68ab08de4f8ff9d6eb8 Diff: https://reviews.apache.org/r/25412/diff/ Testing --- 1. Manual test: with the changes, all following cases are working just like the other command SHOW TABLES... SHOW TABLE EXTENDED IN test_db LIKE 'ba*'; SHOW TABLE EXTENDED IN test_db LIKE ba*; SHOW TABLE EXTENDED IN test_db LIKE `ba*`; 2. six related unit tests are added to show_tables.q and they work as expected 3. submit this patch for pre-commit test to see if the changes will cause other regression. Thanks, Chaoyu Tang
[jira] [Updated] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-1363:
--
Attachment: HIVE-1363.1.patch
Thanks [~xuefuz] for the review. I made the change based on the comments and
uploaded a new patch here.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0, 0.14.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.1.patch, HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7553) avoid the scheduling maintenance window for every jar change
[ https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-7553: --- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Thank you very much [~ferd]!! I have committed this to trunk!! avoid the scheduling maintenance window for every jar change Key: HIVE-7553 URL: https://issues.apache.org/jira/browse/HIVE-7553 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Ferdinand Xu Assignee: Ferdinand Xu Fix For: 0.14.0 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, HIVE-7553.4.patch, HIVE-7553.patch, HIVE-7553.pdf, SessionStateTest.jar.v1, SessionStateTest.jar.v2 When user needs to refresh existing or add a new jar to HS2, it needs to restart it. As HS2 is service exposed to clients, this requires scheduling maintenance window for every jar change. It would be great if we could avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25414: HIVE-7788 Generate plans for insert, update, and delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/#review52542 --- I obivously don't have context here but I do have a few items which I think should be addressed. Thx! common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/25414/#comment91348 I think this should be HIVE_IN_TEZ_TEST ql/src/java/org/apache/hadoop/hive/ql/Context.java https://reviews.apache.org/r/25414/#comment91349 there is a setter/getter for this field so I think it can be private. ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java https://reviews.apache.org/r/25414/#comment91350 assert is almost never enabled. Should we use preconditions? ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java https://reviews.apache.org/r/25414/#comment91340 constants should be all caps. If we fix this one can we fix bucketFileFilter as well. ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java https://reviews.apache.org/r/25414/#comment91342 seems like we might want to log the exception here ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java https://reviews.apache.org/r/25414/#comment91343 seems like we might want to log the exception here ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java https://reviews.apache.org/r/25414/#comment91344 hmm, why not just log as INFO or DEBUG? ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java https://reviews.apache.org/r/25414/#comment91341 no need for stringifyException here, you can pass e as a second arg ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java https://reviews.apache.org/r/25414/#comment91345 looks like this can be final ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java https://reviews.apache.org/r/25414/#comment91346 why not log the exception as well ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java https://reviews.apache.org/r/25414/#comment91347 I think we should change this to IllegalStateException - Brock Noland On Sept. 6, 2014, 4:32 p.m., Alan Gates wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/ --- (Updated Sept. 6, 2014, 4:32 p.m.) Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and Thejas Nair. Bugs: HIVE-7788 https://issues.apache.org/jira/browse/HIVE-7788 Repository: hive-git Description --- This patch adds plan generation as well as making modifications to some of the exec operators to make insert/value, update, and delete work. The patch is large, but about 2/3 of that are tests. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 data/conf/tez/hive-site.xml 0b3877c itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 1a84024 itests/src/test/resources/testconfiguration.properties 99049ca metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java f1697bb ql/src/java/org/apache/hadoop/hive/ql/Context.java 7fcbe3c ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9953919 ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 4246d68 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java f018ca0 ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java e3bc3b1 ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 7f1d71b ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java b1c4441 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 913d3ac ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 264052f ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 8354ad9 ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java 32d2f7a ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2b1a345 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java 96a5d78 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java 5c711cf ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java b5b2b60 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e4a30a2 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 026efe8 ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 2dbf1c8
[jira] [Created] (HIVE-8015) Merge from trunk (3) [Spark Branch]
Brock Noland created HIVE-8015: -- Summary: Merge from trunk (3) [Spark Branch] Key: HIVE-8015 URL: https://issues.apache.org/jira/browse/HIVE-8015 Project: Hive Issue Type: Sub-task Reporter: Brock Noland -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7923) populate stats for test tables
[
https://issues.apache.org/jira/browse/HIVE-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124567#comment-14124567
]
Hive QA commented on HIVE-7923:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666933/HIVE-7923.5.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 6180 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_named_struct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_struct
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join1
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/669/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/669/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-669/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666933
populate stats for test tables
--
Key: HIVE-7923
URL: https://issues.apache.org/jira/browse/HIVE-7923
Project: Hive
Issue Type: Improvement
Reporter: pengcheng xiong
Assignee: pengcheng xiong
Priority: Minor
Attachments: HIVE-7923.1.patch, HIVE-7923.2.patch, HIVE-7923.3.patch,
HIVE-7923.4.patch, HIVE-7923.5.patch
Current q_test only generates tables, e.g., src only but does not create
status. All the test cases will fail in CBO because CBO depends on the
status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8015) Merge from trunk (3) [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-8015: --- Attachment: HIVE-8015.1-spark.patch Merge from trunk (3) [Spark Branch] --- Key: HIVE-8015 URL: https://issues.apache.org/jira/browse/HIVE-8015 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Attachments: HIVE-8015.1-post-merge.patch, HIVE-8015.1-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8015) Merge from trunk (3) [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-8015: --- Assignee: Brock Noland Status: Patch Available (was: Open) Merge from trunk (3) [Spark Branch] --- Key: HIVE-8015 URL: https://issues.apache.org/jira/browse/HIVE-8015 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-8015.1-post-merge.patch, HIVE-8015.1-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8015) Merge from trunk (3) [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-8015: --- Attachment: HIVE-8015.1-post-merge.patch Merge from trunk (3) [Spark Branch] --- Key: HIVE-8015 URL: https://issues.apache.org/jira/browse/HIVE-8015 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-8015.1-post-merge.patch, HIVE-8015.1-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7946) CBO: Merge CBO changes to Trunk
[ https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124578#comment-14124578 ] Laljo John Pullokkaran commented on HIVE-7946: -- Brock, thanks for looking at the patch. We are in the process of cleanup both style and dead code. At this point we are trying to get all the unit tests to pass. CBO: Merge CBO changes to Trunk --- Key: HIVE-7946 URL: https://issues.apache.org/jira/browse/HIVE-7946 Project: Hive Issue Type: Bug Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-7946.1.patch, HIVE-7946.2.patch, HIVE-7946.3.patch, HIVE-7946.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8011) Fix couple of tests to work when FileSystem.access() is available
[
https://issues.apache.org/jira/browse/HIVE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124608#comment-14124608
]
Hive QA commented on HIVE-8011:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666951/HIVE-8011.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6184 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/670/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/670/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-670/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666951
Fix couple of tests to work when FileSystem.access() is available
-
Key: HIVE-8011
URL: https://issues.apache.org/jira/browse/HIVE-8011
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-8011.1.patch
Couple test failures when running against versions of Hadoop that support
FileSystem.access():
TestStorageBasedMetastoreAuthorizationProviderWithACL: Fails with the extra
tests added in HIVE-6093. The parent class does not override
disallowCreateDatabase(), and it looks like this test should not override it
either either.
TestStorageBasedClientSideAuthorizationProvider: Should change the error
message string we're checking against.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6550) SemanticAnalyzer.reset() doesn't clear all the state
[ https://issues.apache.org/jira/browse/HIVE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6550: --- Affects Version/s: 0.13.1 Status: Open (was: Patch Available) Test failures needs to be looked at. SemanticAnalyzer.reset() doesn't clear all the state Key: HIVE-6550 URL: https://issues.apache.org/jira/browse/HIVE-6550 Project: Hive Issue Type: Bug Affects Versions: 0.13.1, 0.13.0, 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Sergey Shelukhin Attachments: HIVE-6550.01.patch, HIVE-6550.02.patch, HIVE-6550.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7923) populate stats for test tables
[ https://issues.apache.org/jira/browse/HIVE-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7923: --- Status: Open (was: Patch Available) Test failures needs to be looked at. populate stats for test tables -- Key: HIVE-7923 URL: https://issues.apache.org/jira/browse/HIVE-7923 Project: Hive Issue Type: Improvement Reporter: pengcheng xiong Assignee: pengcheng xiong Priority: Minor Attachments: HIVE-7923.1.patch, HIVE-7923.2.patch, HIVE-7923.3.patch, HIVE-7923.4.patch, HIVE-7923.5.patch Current q_test only generates tables, e.g., src only but does not create status. All the test cases will fail in CBO because CBO depends on the status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-7813) Hive join key not null shouldn't be generated for partition column
[ https://issues.apache.org/jira/browse/HIVE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-7813. Resolution: Fixed Fix Version/s: 0.14.0 Fixed via HIVE-7912 Hive join key not null shouldn't be generated for partition column -- Key: HIVE-7813 URL: https://issues.apache.org/jira/browse/HIVE-7813 Project: Hive Issue Type: Bug Reporter: Laljo John Pullokkaran Assignee: Ashutosh Chauhan Fix For: 0.14.0 Hive generates not null predicate on join key to avoid scanning in nulls. However for partition columns this is not needed as partition column will always be non null. Without this, PPR may evaluate not null predicate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7813) Hive join key not null shouldn't be generated for partition column
[ https://issues.apache.org/jira/browse/HIVE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7813: --- Affects Version/s: 0.14.0 Hive join key not null shouldn't be generated for partition column -- Key: HIVE-7813 URL: https://issues.apache.org/jira/browse/HIVE-7813 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.14.0 Reporter: Laljo John Pullokkaran Assignee: Ashutosh Chauhan Fix For: 0.14.0 Hive generates not null predicate on join key to avoid scanning in nulls. However for partition columns this is not needed as partition column will always be non null. Without this, PPR may evaluate not null predicate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7813) Hive join key not null shouldn't be generated for partition column
[ https://issues.apache.org/jira/browse/HIVE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7813: --- Component/s: Query Processor Hive join key not null shouldn't be generated for partition column -- Key: HIVE-7813 URL: https://issues.apache.org/jira/browse/HIVE-7813 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.14.0 Reporter: Laljo John Pullokkaran Assignee: Ashutosh Chauhan Fix For: 0.14.0 Hive generates not null predicate on join key to avoid scanning in nulls. However for partition columns this is not needed as partition column will always be non null. Without this, PPR may evaluate not null predicate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8010) [CBO] Handle nested types
[ https://issues.apache.org/jira/browse/HIVE-8010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8010: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to cbo branch. [CBO] Handle nested types - Key: HIVE-8010 URL: https://issues.apache.org/jira/browse/HIVE-8010 Project: Hive Issue Type: Sub-task Components: CBO, Logical Optimizer Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8010-cbo.patch need to handle ExprNodeFieldDesc -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-7967) CBO Trunk Merge: Fall Back in case of complex types
[ https://issues.apache.org/jira/browse/HIVE-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-7967. Resolution: Fixed Fixed via HIVE-8010 CBO Trunk Merge: Fall Back in case of complex types --- Key: HIVE-7967 URL: https://issues.apache.org/jira/browse/HIVE-7967 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Ashutosh Chauhan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7992) StatsRulesProcFactory should gracefully handle overflows
[
https://issues.apache.org/jira/browse/HIVE-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124649#comment-14124649
]
Hive QA commented on HIVE-7992:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666953/HIVE-7992.1.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6184 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/671/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/671/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-671/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666953
StatsRulesProcFactory should gracefully handle overflows
Key: HIVE-7992
URL: https://issues.apache.org/jira/browse/HIVE-7992
Project: Hive
Issue Type: Sub-task
Components: Statistics
Affects Versions: 0.13.1
Reporter: Mostafa Mokhtar
Assignee: Prasanth J
Attachments: HIVE-7992.1.patch
When StatsRulesProcFactory overflows it sets data size to 0 and as a result
the Vertex will ask for a single task, this results in a fairly slow running
query, most likely the overflow is a result of higher than usual number of
rows.
The class should detect an overflow and set a flag when an overflow occurs,
if an overflow occurs StatsRulesProcFactory should request the maximum number
of tasks for the vertex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7969) CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer
[ https://issues.apache.org/jira/browse/HIVE-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7969: - Summary: CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer (was: Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer) CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer -- Key: HIVE-7969 URL: https://issues.apache.org/jira/browse/HIVE-7969 Project: Hive Issue Type: Sub-task Components: CBO, Logical Optimizer Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-7969.patch After patch series of OPTIQ-391 , OPTIQ-392 , OPTIQ-395 , OPTIQ-396 its now possible to use Optiq's native FieldTrimmer. So, lets use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7965) CBO:Handle Row Schema
[ https://issues.apache.org/jira/browse/HIVE-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7965: - Summary: CBO:Handle Row Schema (was: Handle Row Schema) CBO:Handle Row Schema - Key: HIVE-7965 URL: https://issues.apache.org/jira/browse/HIVE-7965 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8013) CBO:fix checkstyle warnings
[ https://issues.apache.org/jira/browse/HIVE-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-8013: - Summary: CBO:fix checkstyle warnings (was: fix checkstyle warnings) CBO:fix checkstyle warnings --- Key: HIVE-8013 URL: https://issues.apache.org/jira/browse/HIVE-8013 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7655) CBO: Reading of partitioned table stats slows down explain
[ https://issues.apache.org/jira/browse/HIVE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7655: - Summary: CBO: Reading of partitioned table stats slows down explain (was: Reading of partitioned table stats slows down explain) CBO: Reading of partitioned table stats slows down explain -- Key: HIVE-7655 URL: https://issues.apache.org/jira/browse/HIVE-7655 Project: Hive Issue Type: Sub-task Affects Versions: 0.13.1 Reporter: Mostafa Mokhtar Assignee: Harish Butani Labels: hive Fix For: 0.14.0 Attachments: HIVE-7655.1.patch This defect is due to a regression introduced in https://issues.apache.org/jira/browse/HIVE-7625, explain for queries that touch partitioned tables is 10x slower. RelOptHiveTable.getRowCount calls listPartitionsWithAuthInfo which returns the data from all partitions, listPartitionsByExpr should be used instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7612) CBO: Add link to parent vertex to mapjoin in explain
[ https://issues.apache.org/jira/browse/HIVE-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7612: - Summary: CBO: Add link to parent vertex to mapjoin in explain (was: Add link to parent vertex to mapjoin in explain) CBO: Add link to parent vertex to mapjoin in explain Key: HIVE-7612 URL: https://issues.apache.org/jira/browse/HIVE-7612 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7612.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7703) CBO: Don't recompute partition list between optiq and hive planning
[ https://issues.apache.org/jira/browse/HIVE-7703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7703: - Summary: CBO: Don't recompute partition list between optiq and hive planning (was: Dont recompute partition list between optiq and hive planning) CBO: Don't recompute partition list between optiq and hive planning --- Key: HIVE-7703 URL: https://issues.apache.org/jira/browse/HIVE-7703 Project: Hive Issue Type: Sub-task Components: CBO, Logical Optimizer Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: h-7703.patch otherwise it will add to latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7463) CBO:Add rule for transitive inference
[ https://issues.apache.org/jira/browse/HIVE-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7463: - Summary: CBO:Add rule for transitive inference (was: Add rule for transitive inference) CBO:Add rule for transitive inference - Key: HIVE-7463 URL: https://issues.apache.org/jira/browse/HIVE-7463 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Harish Butani R1.x=R2.x and R1.x=10 - R2.x = 10 This applies to Inner Joins some form of outer join conditions and filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7565) CBO: Fix exception in Greedy Join reordering Algo
[ https://issues.apache.org/jira/browse/HIVE-7565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7565: - Summary: CBO: Fix exception in Greedy Join reordering Algo (was: Fix exception in Greedy Join reordering Algo) CBO: Fix exception in Greedy Join reordering Algo - Key: HIVE-7565 URL: https://issues.apache.org/jira/browse/HIVE-7565 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-7565.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7546) CBO: Pull partition and column stats regardless of hive conf settings
[ https://issues.apache.org/jira/browse/HIVE-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7546: - Summary: CBO: Pull partition and column stats regardless of hive conf settings (was: Pull partition and column stats regardless of hive conf settings) CBO: Pull partition and column stats regardless of hive conf settings - Key: HIVE-7546 URL: https://issues.apache.org/jira/browse/HIVE-7546 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7546.1.patch Because of performance reasons we have flags to avoid loading certain stats when requested. That's fine for the join selection algo, but CBO needs these stats. This patch separates the two so that CBO can fetch the stats it needs and the regular stats annotation can be controlled via flags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7611) CBO: Workaround for pom issue with optiq-avatica
[ https://issues.apache.org/jira/browse/HIVE-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7611: - Summary: CBO: Workaround for pom issue with optiq-avatica (was: Workaround for pom issue with optiq-avatica) CBO: Workaround for pom issue with optiq-avatica Key: HIVE-7611 URL: https://issues.apache.org/jira/browse/HIVE-7611 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7611.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-7463) CBO:Add rule for transitive inference
[ https://issues.apache.org/jira/browse/HIVE-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran resolved HIVE-7463. -- Resolution: Fixed CBO:Add rule for transitive inference - Key: HIVE-7463 URL: https://issues.apache.org/jira/browse/HIVE-7463 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Harish Butani R1.x=R2.x and R1.x=10 - R2.x = 10 This applies to Inner Joins some form of outer join conditions and filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7395) CBO: Work around non availability of stats for partition columns
[ https://issues.apache.org/jira/browse/HIVE-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7395: - Summary: CBO: Work around non availability of stats for partition columns (was: Work around non availability of stats for partition columns) CBO: Work around non availability of stats for partition columns Key: HIVE-7395 URL: https://issues.apache.org/jira/browse/HIVE-7395 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-7395.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7392) CBO: Support Columns Stats for Partition Columns
[ https://issues.apache.org/jira/browse/HIVE-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7392: - Summary: CBO: Support Columns Stats for Partition Columns (was: Support Columns Stats for Partition Columns) CBO: Support Columns Stats for Partition Columns Key: HIVE-7392 URL: https://issues.apache.org/jira/browse/HIVE-7392 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Ashutosh Chauhan Attachments: h-7392.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7388) CBO: Remove non-ascii char from comments
[ https://issues.apache.org/jira/browse/HIVE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7388: - Summary: CBO: Remove non-ascii char from comments (was: Remove non-ascii char from comments) CBO: Remove non-ascii char from comments Key: HIVE-7388 URL: https://issues.apache.org/jira/browse/HIVE-7388 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7388.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7401) CBO:Fetch Column stats on Demand
[ https://issues.apache.org/jira/browse/HIVE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-7401: - Summary: CBO:Fetch Column stats on Demand (was: Fetch Column stats on Demand) CBO:Fetch Column stats on Demand Key: HIVE-7401 URL: https://issues.apache.org/jira/browse/HIVE-7401 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8016) CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup
Laljo John Pullokkaran created HIVE-8016: Summary: CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup Key: HIVE-8016 URL: https://issues.apache.org/jira/browse/HIVE-8016 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8016) CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup
[ https://issues.apache.org/jira/browse/HIVE-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-8016: - Attachment: HIVE-8016.patch CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup - Key: HIVE-8016 URL: https://issues.apache.org/jira/browse/HIVE-8016 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-8016.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8016) CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup
[ https://issues.apache.org/jira/browse/HIVE-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-8016: - Status: Patch Available (was: Open) CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup - Key: HIVE-8016 URL: https://issues.apache.org/jira/browse/HIVE-8016 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-8016.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25414: HIVE-7788 Generate plans for insert, update, and delete
On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: I obivously don't have context here but I do have a few items which I think should be addressed. Thx! Thanks for the review. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 305 https://reviews.apache.org/r/25414/diff/1/?file=682007#file682007line305 I think this should be HIVE_IN_TEZ_TEST :), will fix On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/Context.java, line 105 https://reviews.apache.org/r/25414/diff/1/?file=682012#file682012line105 there is a setter/getter for this field so I think it can be private. Ok. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java, line 275 https://reviews.apache.org/r/25414/diff/1/?file=682016#file682016line275 assert is almost never enabled. Should we use preconditions? I put this here as a way to test while I was developing, and left it because it helped make clear to later maintainers what I was expecting. I avoided doing an explicit instanceof check for performance. If you think it's important I can put it in there without the assert and then throw an exception. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java, line 52 https://reviews.apache.org/r/25414/diff/1/?file=682019#file682019line52 constants should be all caps. If we fix this one can we fix bucketFileFilter as well. I'm fine to change it, except that all of the other filters in the file aren't, so I was matching existing style. We might want to file a separate JIRA to fix them all, which should be a quick patch. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 2404 https://reviews.apache.org/r/25414/diff/1/?file=682024#file682024line2404 seems like we might want to log the exception here Agreed, will fix. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 2417 https://reviews.apache.org/r/25414/diff/1/?file=682024#file682024line2417 seems like we might want to log the exception here Agreed, will fix. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 2436 https://reviews.apache.org/r/25414/diff/1/?file=682024#file682024line2436 hmm, why not just log as INFO or DEBUG? Will add an INFO message. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 2449 https://reviews.apache.org/r/25414/diff/1/?file=682024#file682024line2449 no need for stringifyException here, you can pass e as a second arg Will fix. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java, line 99 https://reviews.apache.org/r/25414/diff/1/?file=682028#file682028line99 looks like this can be final Sure, but why? On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java, line 747 https://reviews.apache.org/r/25414/diff/1/?file=682029#file682029line747 why not log the exception as well Do you mean the exception stack or the exception name? The exception message is getting logged, since it's in errMsg. I'm throwing a SemanticException that includes the caught exception, so I'm assuming the stack will be printed when that is dumped. On Sept. 6, 2014, 5:43 p.m., Brock Noland wrote: ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java, line 333 https://reviews.apache.org/r/25414/diff/1/?file=682031#file682031line333 I think we should change this to IllegalStateException The dangers of using comedy in your error messages is that you'll forget to go back and put something useful in. Will fix. - Alan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/#review52542 --- On Sept. 6, 2014, 4:32 p.m., Alan Gates wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/ --- (Updated Sept. 6, 2014, 4:32 p.m.) Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and Thejas Nair. Bugs: HIVE-7788 https://issues.apache.org/jira/browse/HIVE-7788 Repository: hive-git Description --- This patch adds plan generation as well as making modifications to some of the exec operators to make insert/value, update, and delete work.
[jira] [Updated] (HIVE-8015) Merge from trunk (3) [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-8015: --- Attachment: (was: HIVE-8015.1-post-merge.patch) Merge from trunk (3) [Spark Branch] --- Key: HIVE-8015 URL: https://issues.apache.org/jira/browse/HIVE-8015 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-8015.1-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124687#comment-14124687
]
Hive QA commented on HIVE-1363:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667028/HIVE-1363.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6184 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table_json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/672/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/672/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-672/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667028
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0, 0.14.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.1.patch, HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8012) TestHiveServer2Concurrency is not implemented
[
https://issues.apache.org/jira/browse/HIVE-8012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124696#comment-14124696
]
Brock Noland commented on HIVE-8012:
Yeah that test was excluded from the old days of the ant build. I never even
looked at it and just excluded it here.
TestHiveServer2Concurrency is not implemented
-
Key: HIVE-8012
URL: https://issues.apache.org/jira/browse/HIVE-8012
Project: Hive
Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere
{code}
@Test
public void test() {
fail(Not yet implemented);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8015) Merge from trunk (3) [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124701#comment-14124701
]
Hive QA commented on HIVE-8015:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667029/HIVE-8015.1-spark.patch
{color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 6338 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_bmj_schema_evolution
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/116/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/116/console
Test logs:
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-116/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 30 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667029
Merge from trunk (3) [Spark Branch]
---
Key: HIVE-8015
URL: https://issues.apache.org/jira/browse/HIVE-8015
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8015.1-spark.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7926) long-lived daemons for query fragment execution, I/O and caching
[ https://issues.apache.org/jira/browse/HIVE-7926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124708#comment-14124708 ] Brock Noland commented on HIVE-7926: bq. Small/short queries can be largely processed by this daemon directly, while any heavy lifting will be performed in standard YARN containers. Can you speak to how this would work? For example, how would a request flow through the system in both cases with LLAP running? How would the decision be made to send a request directly to LLAP? long-lived daemons for query fragment execution, I/O and caching Key: HIVE-7926 URL: https://issues.apache.org/jira/browse/HIVE-7926 Project: Hive Issue Type: New Feature Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: LLAPdesigndocument.pdf We are proposing a new execution model for Hive that is a combination of existing process-based tasks and long-lived daemons running on worker nodes. These nodes can take care of efficient I/O, caching and query fragment execution, while heavy lifting like most joins, ordering, etc. can be handled by tasks. The proposed model is not a 2-system solution for small and large queries; neither it is a separate execution engine like MR or Tez. It can be used by any Hive execution engine, if support is added; in future even external products (e.g. Pig) can use it. The document with high-level design we are proposing will be attached shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7405) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
[
https://issues.apache.org/jira/browse/HIVE-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124718#comment-14124718
]
Hive QA commented on HIVE-7405:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12666959/HIVE-7405.994.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6184 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/673/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/673/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-673/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12666959
Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
--
Key: HIVE-7405
URL: https://issues.apache.org/jira/browse/HIVE-7405
Project: Hive
Issue Type: Sub-task
Components: Vectorization
Reporter: Matt McCline
Assignee: Matt McCline
Attachments: HIVE-7405.1.patch, HIVE-7405.2.patch, HIVE-7405.3.patch,
HIVE-7405.4.patch, HIVE-7405.5.patch, HIVE-7405.6.patch, HIVE-7405.7.patch,
HIVE-7405.8.patch, HIVE-7405.9.patch, HIVE-7405.91.patch, HIVE-7405.92.patch,
HIVE-7405.93.patch, HIVE-7405.94.patch, HIVE-7405.95.patch,
HIVE-7405.96.patch, HIVE-7405.97.patch, HIVE-7405.98.patch,
HIVE-7405.99.patch, HIVE-7405.991.patch, HIVE-7405.994.patch
Vectorize the basic case that does not have any count distinct aggregation.
Add a 4th processing mode in VectorGroupByOperator for reduce where each
input VectorizedRowBatch has only values for one key at a time. Thus, the
values in the batch can be aggregated quickly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-1363:
--
Attachment: HIVE-1363.2.patch
The describe_table_json.q test failure is related to the change from this
patch. Actually the original test output seemed not right. For the query SHOW
TABLE EXTENDED LIKE 'json*', it returned empty results and its output in json
was {tables:[]}. But the expected result should have one entry for table
jsontable, and the output should look like following, which is to be masked in
its q.out file.
==
{tables:[{minFileSize:0,totalNumberFiles:0,location:file:/user/hive/warehouse/apache/jsontable,outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,lastAccessTime:0,lastUpdateTime:1410049821000,columns:[{name:key,type:int},{name:value,type:string}],maxFileSize:0,partitioned:false,tableName:jsontable,owner:ctang,inputFormat:org.apache.hadoop.mapred.TextInputFormat,totalFileSize:0}]}
==
Change describe_table_json.q.out to reflect the expected query output and
uploaded a new patch.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0, 0.14.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.1.patch, HIVE-1363.2.patch, HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-2390) Expand support for union types
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124734#comment-14124734
]
Hive QA commented on HIVE-2390:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667013/HIVE-2390.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6184 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/674/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/674/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-674/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667013
Expand support for union types
--
Key: HIVE-2390
URL: https://issues.apache.org/jira/browse/HIVE-2390
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Jakob Homan
Assignee: Suma Shivaprasad
Labels: uniontype
Fix For: 0.14.0
Attachments: HIVE-2390.1.patch, HIVE-2390.patch
When the union type was introduced, full support for it wasn't provided. For
instance, when working with a union that gets passed to LazyBinarySerde:
{noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: hive unit test report question
hi all: I would like to create a jenkins job to run both hive ut and integration test. Right now it seems that I have to execute mulitple maven goals in different poms: mvn clean install surefire-report:report -Daggregate=true -Phadoop-2 cd itests mvn clean install surefire-report:report -Daggregate=true -Phadoop-2 I would like to use one maven jenkins job and right now I cant figure out how to configure job propery to execute maven goals in different poms (maybe I can add post-build step to execute another shell?). Each hive ptest2 job can run all tests and I would like to know the configure it use. Any help is appreciated. Thanks. 2014-01-14 14:05 GMT+08:00 Shanyu Zhao shz...@microsoft.com: Thanks guys for your help! I found Eugene's comments are particularly helpful. With -Daggregate=true I now can see an aggregated unit test results. Btw, I didn't mean to run itests, I just want to run all unit tests. I think in the FAQ they made it clear that itests are disconnected from the top level pom.xml. Shanyu -Original Message- From: Eugene Koifman [mailto:ekoif...@hortonworks.com] Sent: Monday, January 13, 2014 4:06 PM To: dev@hive.apache.org Subject: Re: hive unit test report question I think you want to add -Daggregate=true you should then have target/site/surefire-report.html in the module where you ran the command On Mon, Jan 13, 2014 at 2:54 PM, Szehon Ho sze...@cloudera.com wrote: Hi Shanyu, Are you running in /itests? The unit tests are in there, and are not run if you are running from the root. Thanks Szehon On Mon, Jan 13, 2014 at 1:59 PM, Shanyu Zhao shz...@microsoft.com wrote: Hi, I was trying to build hive trunk, run all unit tests and generate reports, but I'm not sure what's the correct command line. I was using: mvn clean install -Phadoop-2 -DskipTests mvn test surefire-report:report -Phadoop-2 But the reports in the root folder and several other projects (such as metastore) are empty with no test results. And I couldn't find a summary page for all unit tests. I was trying to avoid mvn site because it seems to take forever to finish. Am I using the correct commands? How can I get a report like the one in the precommit report: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/827/testRep ort/ ? I really appreciate your help! Shanyu -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (HIVE-7949) Create table LIKE command doesn't set new owner
[ https://issues.apache.org/jira/browse/HIVE-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124740#comment-14124740 ] Lefty Leverenz commented on HIVE-7949: -- This is a regression from 0.12 Should this bug in 0.13.0 0.13.1 be documented in the wiki? * [DDL -- Create Table Like | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableLike] Create table LIKE command doesn't set new owner --- Key: HIVE-7949 URL: https://issues.apache.org/jira/browse/HIVE-7949 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.13.1 Reporter: Pala M Muthaia Assignee: Pala M Muthaia Fix For: 0.14.0 Attachments: HIVE-7949.1.patch 'Create table like' command doesn't set the current user as owner of new table, instead new table owner is same as source table owner. This is a regression from 0.12 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8017) Use HiveKey instead of Byteswritable as key type of the pair RDD [Spark Branch]
Rui Li created HIVE-8017:
Summary: Use HiveKey instead of Byteswritable as key type of the
pair RDD [Spark Branch]
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases, we
have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join, bucketed
table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8017) Use HiveKey instead of Byteswritable as key type of the pair RDD [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-8017:
-
Issue Type: Sub-task (was: Bug)
Parent: HIVE-7292
Use HiveKey instead of Byteswritable as key type of the pair RDD [Spark
Branch]
---
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases,
we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join,
bucketed table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8017) Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-8017:
-
Summary: Use HiveKey instead of BytesWritable as key type of the pair RDD
[Spark Branch] (was: Use HiveKey instead of Byteswritable as key type of the
pair RDD [Spark Branch])
Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark
Branch]
---
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases,
we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join,
bucketed table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7956) When inserting into a bucketed table, all data goes to a single bucket [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124742#comment-14124742
]
Rui Li commented on HIVE-7956:
--
[~xuefuz] that's great! I created HIVE-8017 and will do the refactor under that.
When inserting into a bucketed table, all data goes to a single bucket [Spark
Branch]
-
Key: HIVE-7956
URL: https://issues.apache.org/jira/browse/HIVE-7956
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
I created a bucketed table:
{code}
create table testBucket(x int,y string) clustered by(x) into 10 buckets;
{code}
Then I run a query like:
{code}
set hive.enforce.bucketing = true;
insert overwrite table testBucket select intCol,stringCol from src;
{code}
Here {{src}} is a simple textfile-based table containing 4000 records
(not bucketed). The query launches 10 reduce tasks but all the data goes to
only one of them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8008) NPE while reading null decimal value
[
https://issues.apache.org/jira/browse/HIVE-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124756#comment-14124756
]
Xuefu Zhang commented on HIVE-8008:
---
+1. I think the above test failure isn't related.
NPE while reading null decimal value
Key: HIVE-8008
URL: https://issues.apache.org/jira/browse/HIVE-8008
Project: Hive
Issue Type: Bug
Reporter: Chao
Assignee: Chao
Attachments: HIVE-8008.2.patch, HIVE-8008.3.patch, HIVE-8008.4.patch,
HIVE-8008.patch
Say you have this table {{dec_test}}:
{code}
dec decimal(10,0)
{code}
If the table has a row that is 99.5, and if we do
{code}
select * from dec_test;
{code}
it will crash with NPE:
{code}
2014-09-05 14:08:56,023 ERROR [main]: CliDriver
(SessionState.java:printError(545)) - Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:151)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1531)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:285)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:544)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:536)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:137)
... 12 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:265)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:71)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:70)
at
org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:39)
at
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:87)
... 19 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124760#comment-14124760
]
Xuefu Zhang commented on HIVE-1363:
---
+1, pending on the new test result.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Affects Versions: 0.5.0, 0.14.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.1.patch, HIVE-1363.2.patch, HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7946) CBO: Merge CBO changes to Trunk
[
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124763#comment-14124763
]
Hive QA commented on HIVE-7946:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667018/HIVE-7946.3.patch
{color:red}ERROR:{color} -1 due to 350 failed/errored test(s), 5556 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver_accumulo_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_view_sqlstd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_partitioned_native
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_correctness
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_cast
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_udf1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_struct_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_union_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_distinct_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_04_evolved_parts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explode_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_distinct_samekey
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_id1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_id2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets1
Re: Review Request 25176: HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/#review52554 --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java https://reviews.apache.org/r/25176/#comment91373 That's weird. I can see that they are indeed not consistent. Thanks for the explanation. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java https://reviews.apache.org/r/25176/#comment91372 sorry for my curiosity. If the dir is the same, wouldn't data overwrites each other? - Xuefu Zhang On Sept. 4, 2014, 5:03 p.m., Na Yang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/ --- (Updated Sept. 4, 2014, 5:03 p.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7870 https://issues.apache.org/jira/browse/HIVE-7870 Repository: hive-git Description --- HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch] The cause of this problem is during spark/tez task generation, the union file sink operator are cloned to two new filesink operator. The linkedfilesinkdesc info for those new filesink operators are missing. In addition, the two new filesink operators also need to be linked together. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 5ddc16d ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 379a39c ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 76fc290 ql/src/test/queries/clientpositive/union_remove_1.q c87b3fe ql/src/test/queries/clientpositive/union_remove_10.q 6701952 ql/src/test/queries/clientpositive/union_remove_11.q 4b2fa42 ql/src/test/queries/clientpositive/union_remove_12.q 69d0d0a ql/src/test/queries/clientpositive/union_remove_13.q 7605f0e ql/src/test/queries/clientpositive/union_remove_14.q a4fdfc8 ql/src/test/queries/clientpositive/union_remove_15.q e3c937b ql/src/test/queries/clientpositive/union_remove_16.q 537078b ql/src/test/queries/clientpositive/union_remove_17.q d70f3d3 ql/src/test/queries/clientpositive/union_remove_18.q 6352bc3 ql/src/test/queries/clientpositive/union_remove_19.q 8c45953 ql/src/test/queries/clientpositive/union_remove_2.q 83cd288 ql/src/test/queries/clientpositive/union_remove_20.q f80f7c1 ql/src/test/queries/clientpositive/union_remove_21.q 8963c25 ql/src/test/queries/clientpositive/union_remove_22.q b0c1ccd ql/src/test/queries/clientpositive/union_remove_23.q a1b989a ql/src/test/queries/clientpositive/union_remove_24.q ec561e0 ql/src/test/queries/clientpositive/union_remove_25.q 76c1ff5 ql/src/test/queries/clientpositive/union_remove_3.q 9617f73 ql/src/test/queries/clientpositive/union_remove_4.q cae323b ql/src/test/queries/clientpositive/union_remove_5.q 5df84e1 ql/src/test/queries/clientpositive/union_remove_6.q bfce26d ql/src/test/queries/clientpositive/union_remove_7.q 3a95674 ql/src/test/queries/clientpositive/union_remove_8.q a83a43e ql/src/test/queries/clientpositive/union_remove_9.q e71f6dd ql/src/test/results/clientpositive/spark/union10.q.out 20c681e ql/src/test/results/clientpositive/spark/union18.q.out 3f37a0a ql/src/test/results/clientpositive/spark/union19.q.out 6922fcd ql/src/test/results/clientpositive/spark/union28.q.out 8bd5218 ql/src/test/results/clientpositive/spark/union29.q.out b9546ef ql/src/test/results/clientpositive/spark/union3.q.out 3ae6536 ql/src/test/results/clientpositive/spark/union30.q.out 12717a1 ql/src/test/results/clientpositive/spark/union33.q.out b89757f ql/src/test/results/clientpositive/spark/union4.q.out 6341cd9 ql/src/test/results/clientpositive/spark/union6.q.out 263d9f4 ql/src/test/results/clientpositive/spark/union_remove_10.q.out 927a15d ql/src/test/results/clientpositive/spark/union_remove_11.q.out 96651e1 ql/src/test/results/clientpositive/spark/union_remove_16.q.out 0954ae4 ql/src/test/results/clientpositive/spark/union_remove_4.q.out cc46dda ql/src/test/results/clientpositive/spark/union_remove_5.q.out f6cdeb3 ql/src/test/results/clientpositive/spark/union_remove_9.q.out 1f0260c Diff: https://reviews.apache.org/r/25176/diff/ Testing --- Thanks, Na Yang
Re: Review Request 25176: HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch]
On Sept. 7, 2014, 4:46 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line 1749 https://reviews.apache.org/r/25176/diff/4/?file=676862#file676862line1749 sorry for my curiosity. If the dir is the same, wouldn't data overwrites each other? The data will not overwrite each other in this case. From the .q tests result, it shows the query results are correct. - Na --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/#review52554 --- On Sept. 4, 2014, 5:03 p.m., Na Yang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/ --- (Updated Sept. 4, 2014, 5:03 p.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7870 https://issues.apache.org/jira/browse/HIVE-7870 Repository: hive-git Description --- HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch] The cause of this problem is during spark/tez task generation, the union file sink operator are cloned to two new filesink operator. The linkedfilesinkdesc info for those new filesink operators are missing. In addition, the two new filesink operators also need to be linked together. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 5ddc16d ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 379a39c ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 76fc290 ql/src/test/queries/clientpositive/union_remove_1.q c87b3fe ql/src/test/queries/clientpositive/union_remove_10.q 6701952 ql/src/test/queries/clientpositive/union_remove_11.q 4b2fa42 ql/src/test/queries/clientpositive/union_remove_12.q 69d0d0a ql/src/test/queries/clientpositive/union_remove_13.q 7605f0e ql/src/test/queries/clientpositive/union_remove_14.q a4fdfc8 ql/src/test/queries/clientpositive/union_remove_15.q e3c937b ql/src/test/queries/clientpositive/union_remove_16.q 537078b ql/src/test/queries/clientpositive/union_remove_17.q d70f3d3 ql/src/test/queries/clientpositive/union_remove_18.q 6352bc3 ql/src/test/queries/clientpositive/union_remove_19.q 8c45953 ql/src/test/queries/clientpositive/union_remove_2.q 83cd288 ql/src/test/queries/clientpositive/union_remove_20.q f80f7c1 ql/src/test/queries/clientpositive/union_remove_21.q 8963c25 ql/src/test/queries/clientpositive/union_remove_22.q b0c1ccd ql/src/test/queries/clientpositive/union_remove_23.q a1b989a ql/src/test/queries/clientpositive/union_remove_24.q ec561e0 ql/src/test/queries/clientpositive/union_remove_25.q 76c1ff5 ql/src/test/queries/clientpositive/union_remove_3.q 9617f73 ql/src/test/queries/clientpositive/union_remove_4.q cae323b ql/src/test/queries/clientpositive/union_remove_5.q 5df84e1 ql/src/test/queries/clientpositive/union_remove_6.q bfce26d ql/src/test/queries/clientpositive/union_remove_7.q 3a95674 ql/src/test/queries/clientpositive/union_remove_8.q a83a43e ql/src/test/queries/clientpositive/union_remove_9.q e71f6dd ql/src/test/results/clientpositive/spark/union10.q.out 20c681e ql/src/test/results/clientpositive/spark/union18.q.out 3f37a0a ql/src/test/results/clientpositive/spark/union19.q.out 6922fcd ql/src/test/results/clientpositive/spark/union28.q.out 8bd5218 ql/src/test/results/clientpositive/spark/union29.q.out b9546ef ql/src/test/results/clientpositive/spark/union3.q.out 3ae6536 ql/src/test/results/clientpositive/spark/union30.q.out 12717a1 ql/src/test/results/clientpositive/spark/union33.q.out b89757f ql/src/test/results/clientpositive/spark/union4.q.out 6341cd9 ql/src/test/results/clientpositive/spark/union6.q.out 263d9f4 ql/src/test/results/clientpositive/spark/union_remove_10.q.out 927a15d ql/src/test/results/clientpositive/spark/union_remove_11.q.out 96651e1 ql/src/test/results/clientpositive/spark/union_remove_16.q.out 0954ae4 ql/src/test/results/clientpositive/spark/union_remove_4.q.out cc46dda ql/src/test/results/clientpositive/spark/union_remove_5.q.out f6cdeb3 ql/src/test/results/clientpositive/spark/union_remove_9.q.out 1f0260c Diff: https://reviews.apache.org/r/25176/diff/ Testing --- Thanks, Na Yang
