[jira] [Updated] (HIVE-7870) Insert overwrite table query does not generate correct task plan
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao updated HIVE-7870:
---
Status: Patch Available (was: Open)
Insert overwrite table query does not generate correct task plan
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Chao
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 24688: parallel order by clause on a string column fails with IOException: Split points are out of order
On Aug. 28, 2014, 5:55 a.m., Szehon Ho wrote:
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 1040
https://reviews.apache.org/r/24688/diff/3/?file=669965#file669965line1040
Do you think this makes more sense to user as a absolute number? It's
hard for user to figure out 'expected number of reducer', right?
Yes, expected number of reducer is a value decided by hive internally, which
can be not expected one by user. Would it better to use absolute number than
ratio?
On Aug. 28, 2014, 5:55 a.m., Szehon Ho wrote:
ql/src/test/org/apache/hadoop/hive/ql/exec/TestPartitionKeySampler.java,
line 1
https://reviews.apache.org/r/24688/diff/3/?file=669970#file669970line1
Please add license header.
Ah, thanks.
- Navis
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24688/#review51744
---
On Aug. 27, 2014, 2:18 a.m., Navis Ryu wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24688/
---
(Updated Aug. 27, 2014, 2:18 a.m.)
Review request for hive.
Bugs: HIVE-7669
https://issues.apache.org/jira/browse/HIVE-7669
Repository: hive-git
Description
---
The source table has 600 Million rows and it has a String column
l_shipinstruct which has 4 unique values. (Ie. these 4 values are repeated
across the 600 million rows)
We are sorting it based on this string column l_shipinstruct as shown in
the below HiveQL with the following parameters.
{code:sql}
set hive.optimize.sampling.orderby=true;
set hive.optimize.sampling.orderby.number=1000;
set hive.optimize.sampling.orderby.percent=0.1f;
insert overwrite table lineitem_temp_report
select
l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity,
l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate,
l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment
from
lineitem
order by l_shipinstruct;
{code}
Stack Trace
Diagnostic Messages for this Task:
{noformat}
Error: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.init(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 10 more
Caused by: java.lang.IllegalArgumentException: Can't read partitions file
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:42)
at
org.apache.hadoop.hive.ql.exec.HiveTotalOrderPartitioner.configure(HiveTotalOrderPartitioner.java:37)
... 15 more
Caused by: java.io.IOException: Split points are out of order
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:96)
... 17 more
{noformat}
Diffs
-
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9
common/src/java/org/apache/hadoop/hive/conf/Validator.java cea9c41
ql/src/java/org/apache/hadoop/hive/ql/exec/HiveTotalOrderPartitioner.java
6c22362
ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 166461a
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ef72039
ql/src/test/org/apache/hadoop/hive/ql/exec/TestPartitionKeySampler.java
PRE-CREATION
Diff: https://reviews.apache.org/r/24688/diff/
Testing
---
Thanks,
Navis Ryu
Re: Review Request 24688: parallel order by clause on a string column fails with IOException: Split points are out of order
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24688/#review51747
---
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
https://reviews.apache.org/r/24688/#comment90321
Yep, thats what I meant.
- Szehon Ho
On Aug. 27, 2014, 2:18 a.m., Navis Ryu wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24688/
---
(Updated Aug. 27, 2014, 2:18 a.m.)
Review request for hive.
Bugs: HIVE-7669
https://issues.apache.org/jira/browse/HIVE-7669
Repository: hive-git
Description
---
The source table has 600 Million rows and it has a String column
l_shipinstruct which has 4 unique values. (Ie. these 4 values are repeated
across the 600 million rows)
We are sorting it based on this string column l_shipinstruct as shown in
the below HiveQL with the following parameters.
{code:sql}
set hive.optimize.sampling.orderby=true;
set hive.optimize.sampling.orderby.number=1000;
set hive.optimize.sampling.orderby.percent=0.1f;
insert overwrite table lineitem_temp_report
select
l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity,
l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate,
l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment
from
lineitem
order by l_shipinstruct;
{code}
Stack Trace
Diagnostic Messages for this Task:
{noformat}
Error: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.init(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 10 more
Caused by: java.lang.IllegalArgumentException: Can't read partitions file
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:42)
at
org.apache.hadoop.hive.ql.exec.HiveTotalOrderPartitioner.configure(HiveTotalOrderPartitioner.java:37)
... 15 more
Caused by: java.io.IOException: Split points are out of order
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:96)
... 17 more
{noformat}
Diffs
-
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9
common/src/java/org/apache/hadoop/hive/conf/Validator.java cea9c41
ql/src/java/org/apache/hadoop/hive/ql/exec/HiveTotalOrderPartitioner.java
6c22362
ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 166461a
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ef72039
ql/src/test/org/apache/hadoop/hive/ql/exec/TestPartitionKeySampler.java
PRE-CREATION
Diff: https://reviews.apache.org/r/24688/diff/
Testing
---
Thanks,
Navis Ryu
[jira] [Updated] (HIVE-7777) add CSV support for Serde
[ https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry Ogrodnek updated HIVE-: - Attachment: csv-serde-master.zip Attached is the csv-serde source to the project referenced. It has already been released under the Apache 2 License. thanks. add CSV support for Serde - Key: HIVE- URL: https://issues.apache.org/jira/browse/HIVE- Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: csv-serde-master.zip There is no official support for csvSerde for hive while there is an open source project in github(https://github.com/ogrodnek/csv-serde). CSV is of high frequency in use as a data format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7353) HiveServer2 using embedded MetaStore leaks JDOPersistanceManager
[ https://issues.apache.org/jira/browse/HIVE-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-7353: - Labels: TODOC14 (was: ) HiveServer2 using embedded MetaStore leaks JDOPersistanceManager Key: HIVE-7353 URL: https://issues.apache.org/jira/browse/HIVE-7353 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-7353.1.patch, HIVE-7353.2.patch, HIVE-7353.3.patch, HIVE-7353.4.patch, HIVE-7353.5.patch, HIVE-7353.6.patch, HIVE-7353.7.patch, HIVE-7353.8.patch, HIVE-7353.9.patch While using embedded metastore, while creating background threads to run async operations, HiveServer2 ends up creating new instances of JDOPersistanceManager which are cached in JDOPersistanceManagerFactory. Even when the background thread is killed by the thread pool manager, the JDOPersistanceManager are never GCed because they are cached by JDOPersistanceManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7353) HiveServer2 using embedded MetaStore leaks JDOPersistanceManager
[ https://issues.apache.org/jira/browse/HIVE-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113416#comment-14113416 ] Lefty Leverenz commented on HIVE-7353: -- This adds configuration parameters *hive.server2.thrift.worker.keepalive.time* and *hive.server2.thrift.http.worker.keepalive.time* so they need to be documented in the wiki, preferably after these related parameters: * [Configuration Properties -- HiveServer2 -- hive.server2.thrift.max.worker.threads | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.thrift.max.worker.threads] * [Configuration Properties -- HiveServer2 -- hive.server2.thrift.http.max.worker.threads | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.thrift.http.max.worker.threads] HiveServer2 using embedded MetaStore leaks JDOPersistanceManager Key: HIVE-7353 URL: https://issues.apache.org/jira/browse/HIVE-7353 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-7353.1.patch, HIVE-7353.2.patch, HIVE-7353.3.patch, HIVE-7353.4.patch, HIVE-7353.5.patch, HIVE-7353.6.patch, HIVE-7353.7.patch, HIVE-7353.8.patch, HIVE-7353.9.patch While using embedded metastore, while creating background threads to run async operations, HiveServer2 ends up creating new instances of JDOPersistanceManager which are cached in JDOPersistanceManagerFactory. Even when the background thread is killed by the thread pool manager, the JDOPersistanceManager are never GCed because they are cached by JDOPersistanceManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 21679: HIVE-6245: HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21679/#review51256 --- itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java https://reviews.apache.org/r/21679/#comment89357 please use a name that better indicates that it is a boolean. eg isRemoteMetastore itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java https://reviews.apache.org/r/21679/#comment89358 this status is unused - Thejas Nair On Aug. 20, 2014, 10:33 p.m., Venki Korukanti wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21679/ --- (Updated Aug. 20, 2014, 10:33 p.m.) Review request for hive, Navis Ryu, Thejas Nair, and Vaibhav Gumashta. Bugs: HIVE-6245 https://issues.apache.org/jira/browse/HIVE-6245 Repository: hive-git Description --- Instead of creating Hive object when authentication type is KERBEROS, always create the Hive object in HiveSessionImplWithUGI. HiveSessionImplWithUGI is created when user impersonation is needed. Diffs - itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 4502439 itests/hive-unit/src/test/java/org/apache/hive/service/TestHS2ImpersonationWithRemoteMS.java PRE-CREATION metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 51c3f2c service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 39d2184 Diff: https://reviews.apache.org/r/21679/diff/ Testing --- Patch includes a test. Thanks, Venki Korukanti
[jira] [Commented] (HIVE-7405) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
[
https://issues.apache.org/jira/browse/HIVE-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113436#comment-14113436
]
Hive QA commented on HIVE-7405:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664784/HIVE-7405.91.patch
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 6126 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_vectorization_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_12
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_13
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_15
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_casts
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_nested_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/539/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/539/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-539/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12664784
Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
--
Key: HIVE-7405
URL: https://issues.apache.org/jira/browse/HIVE-7405
Project: Hive
Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline
Attachments: HIVE-7405.1.patch, HIVE-7405.2.patch, HIVE-7405.3.patch,
HIVE-7405.4.patch, HIVE-7405.5.patch, HIVE-7405.6.patch, HIVE-7405.7.patch,
HIVE-7405.8.patch, HIVE-7405.9.patch, HIVE-7405.91.patch
Vectorize the basic case that does not have any count distinct aggregation.
Add a 4th processing mode in VectorGroupByOperator for reduce where each
input VectorizedRowBatch has only values for one key at a time. Thus, the
values in the batch can be aggregated quickly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 24688: parallel order by clause on a string column fails with IOException: Split points are out of order
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24688/#review51752
---
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
https://reviews.apache.org/r/24688/#comment90328
Thank you for saying Probability instead of Percent in the description!
I've been noticing that error in other parameters lately. And since there's
already a Hive tradition of calling parameters xxx.percent when the value is
a probability, I won't request changing the parameter name.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
https://reviews.apache.org/r/24688/#comment90327
Would be good to specify how sampling is enabled, such as If sampling is
enabled with hive.optimize.sampling.orderby, or If sampling is enabled
(hive.optimize.sampling.orderby=true),
- Lefty Leverenz
On Aug. 27, 2014, 2:18 a.m., Navis Ryu wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24688/
---
(Updated Aug. 27, 2014, 2:18 a.m.)
Review request for hive.
Bugs: HIVE-7669
https://issues.apache.org/jira/browse/HIVE-7669
Repository: hive-git
Description
---
The source table has 600 Million rows and it has a String column
l_shipinstruct which has 4 unique values. (Ie. these 4 values are repeated
across the 600 million rows)
We are sorting it based on this string column l_shipinstruct as shown in
the below HiveQL with the following parameters.
{code:sql}
set hive.optimize.sampling.orderby=true;
set hive.optimize.sampling.orderby.number=1000;
set hive.optimize.sampling.orderby.percent=0.1f;
insert overwrite table lineitem_temp_report
select
l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity,
l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate,
l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment
from
lineitem
order by l_shipinstruct;
{code}
Stack Trace
Diagnostic Messages for this Task:
{noformat}
Error: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.init(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 10 more
Caused by: java.lang.IllegalArgumentException: Can't read partitions file
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:42)
at
org.apache.hadoop.hive.ql.exec.HiveTotalOrderPartitioner.configure(HiveTotalOrderPartitioner.java:37)
... 15 more
Caused by: java.io.IOException: Split points are out of order
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:96)
... 17 more
{noformat}
Diffs
-
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9
common/src/java/org/apache/hadoop/hive/conf/Validator.java cea9c41
ql/src/java/org/apache/hadoop/hive/ql/exec/HiveTotalOrderPartitioner.java
6c22362
ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 166461a
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ef72039
ql/src/test/org/apache/hadoop/hive/ql/exec/TestPartitionKeySampler.java
PRE-CREATION
Diff: https://reviews.apache.org/r/24688/diff/
Testing
---
Thanks,
Navis Ryu
[jira] [Commented] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[ https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113443#comment-14113443 ] Thejas M Nair commented on HIVE-6245: - Looks good. I just have some minor comments, added them in reviewboard. HS2 creates DBs/Tables with wrong ownership when HMS setugi is true --- Key: HIVE-6245 URL: https://issues.apache.org/jira/browse/HIVE-6245 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0, 0.13.0 Reporter: Chaoyu Tang Assignee: Venki Korukanti Attachments: HIVE-6245.2.patch.txt, HIVE-6245.3.patch.txt, HIVE-6245.4.patch, HIVE-6245.patch The case with following settings is valid but does not work correctly in current HS2: == hive.server2.authentication=NONE (or LDAP) hive.server2.enable.doAs= true hive.metastore.sasl.enabled=false hive.metastore.execute.setugi=true == Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC application) and create DBs/Tables with user's ownership. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7669) parallel order by clause on a string column fails with IOException: Split points are out of order
[
https://issues.apache.org/jira/browse/HIVE-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113444#comment-14113444
]
Lefty Leverenz commented on HIVE-7669:
--
Thanks for asking, [~szehon]. I had already glanced at the parameter
descriptions and thought they were okay, but closer inspection produced an
improvement request which I've posted on the review board.
General comment: Sampling an order by clause mystifies me -- I don't see how
it relates to parallel execution if overall sampling is not desired. Could we
have a release note explaining it?
parallel order by clause on a string column fails with IOException: Split
points are out of order
-
Key: HIVE-7669
URL: https://issues.apache.org/jira/browse/HIVE-7669
Project: Hive
Issue Type: Bug
Components: HiveServer2, Query Processor, SQL
Affects Versions: 0.12.0
Environment: Hive 0.12.0-cdh5.0.0
OS: Redhat linux
Reporter: Vishal Kamath
Assignee: Navis
Labels: orderby
Attachments: HIVE-7669.1.patch.txt, HIVE-7669.2.patch.txt,
HIVE-7669.3.patch.txt
The source table has 600 Million rows and it has a String column
l_shipinstruct which has 4 unique values. (Ie. these 4 values are repeated
across the 600 million rows)
We are sorting it based on this string column l_shipinstruct as shown in
the below HiveQL with the following parameters.
{code:sql}
set hive.optimize.sampling.orderby=true;
set hive.optimize.sampling.orderby.number=1000;
set hive.optimize.sampling.orderby.percent=0.1f;
insert overwrite table lineitem_temp_report
select
l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity,
l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate,
l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment
from
lineitem
order by l_shipinstruct;
{code}
Stack Trace
Diagnostic Messages for this Task:
{noformat}
Error: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.init(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 10 more
Caused by: java.lang.IllegalArgumentException: Can't read partitions file
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:42)
at
org.apache.hadoop.hive.ql.exec.HiveTotalOrderPartitioner.configure(HiveTotalOrderPartitioner.java:37)
... 15 more
Caused by: java.io.IOException: Split points are out of order
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:96)
... 17 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7870) Insert overwrite table query does not generate correct task plan
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113458#comment-14113458
]
Hive QA commented on HIVE-7870:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664791/HIVE-7870.1-spark.patch
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 6265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_9
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/100/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/100/console
Test logs:
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-100/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12664791
Insert overwrite table query does not generate correct task plan
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Chao
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 21679: HIVE-6245: HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21679/ --- (Updated Aug. 28, 2014, 7:25 a.m.) Review request for hive, Navis Ryu, Thejas Nair, and Vaibhav Gumashta. Changes --- Addressed review comments. Bugs: HIVE-6245 https://issues.apache.org/jira/browse/HIVE-6245 Repository: hive-git Description --- Instead of creating Hive object when authentication type is KERBEROS, always create the Hive object in HiveSessionImplWithUGI. HiveSessionImplWithUGI is created when user impersonation is needed. Diffs (updated) - itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 4502439 itests/hive-unit/src/test/java/org/apache/hive/service/TestHS2ImpersonationWithRemoteMS.java PRE-CREATION metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 51c3f2c service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 39d2184 Diff: https://reviews.apache.org/r/21679/diff/ Testing --- Patch includes a test. Thanks, Venki Korukanti
[jira] [Updated] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[ https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venki Korukanti updated HIVE-6245: -- Attachment: HIVE-6245.5.patch Attaching patch v5. Addressed review comments. HS2 creates DBs/Tables with wrong ownership when HMS setugi is true --- Key: HIVE-6245 URL: https://issues.apache.org/jira/browse/HIVE-6245 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0, 0.13.0 Reporter: Chaoyu Tang Assignee: Venki Korukanti Attachments: HIVE-6245.2.patch.txt, HIVE-6245.3.patch.txt, HIVE-6245.4.patch, HIVE-6245.5.patch, HIVE-6245.patch The case with following settings is valid but does not work correctly in current HS2: == hive.server2.authentication=NONE (or LDAP) hive.server2.enable.doAs= true hive.metastore.sasl.enabled=false hive.metastore.execute.setugi=true == Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC application) and create DBs/Tables with user's ownership. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7557) When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
[
https://issues.apache.org/jira/browse/HIVE-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-7557:
---
Attachment: HIVE-7557.1.patch
When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
---
Key: HIVE-7557
URL: https://issues.apache.org/jira/browse/HIVE-7557
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Rajesh Balamohan
Attachments: HIVE-7557.1.patch
Turned off dynpart_sort_opt_vectorization.q (Tez) since it fails when reduce
is vectorized to get HIVE-7029 checked in.
Stack trace:
{code}
Container released by application,
AttemptID:attempt_1406747677386_0003_2_00_00_2 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:382)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
... 6 more
Caused by:
[jira] [Commented] (HIVE-7557) When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
[
https://issues.apache.org/jira/browse/HIVE-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113478#comment-14113478
]
Matt McCline commented on HIVE-7557:
Patch #1 temporarily turns off vectorization if VectorFileSink would need to
handle dynamic partitions. This gives time to understand what is going on and
fix VectorFileSink.
When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
---
Key: HIVE-7557
URL: https://issues.apache.org/jira/browse/HIVE-7557
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Rajesh Balamohan
Attachments: HIVE-7557.1.patch
Turned off dynpart_sort_opt_vectorization.q (Tez) since it fails when reduce
is vectorized to get HIVE-7029 checked in.
Stack trace:
{code}
Container released by application,
AttemptID:attempt_1406747677386_0003_2_00_00_2 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:382)
at
[jira] [Updated] (HIVE-7557) When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
[
https://issues.apache.org/jira/browse/HIVE-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-7557:
---
Assignee: Matt McCline (was: Rajesh Balamohan)
Status: Patch Available (was: Open)
When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
---
Key: HIVE-7557
URL: https://issues.apache.org/jira/browse/HIVE-7557
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline
Attachments: HIVE-7557.1.patch
Turned off dynpart_sort_opt_vectorization.q (Tez) since it fails when reduce
is vectorized to get HIVE-7029 checked in.
Stack trace:
{code}
Container released by application,
AttemptID:attempt_1406747677386_0003_2_00_00_2 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:382)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
[jira] [Commented] (HIVE-7557) When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
[
https://issues.apache.org/jira/browse/HIVE-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113489#comment-14113489
]
Jitendra Nath Pandey commented on HIVE-7557:
+1. lgtm
When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
---
Key: HIVE-7557
URL: https://issues.apache.org/jira/browse/HIVE-7557
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline
Attachments: HIVE-7557.1.patch
Turned off dynpart_sort_opt_vectorization.q (Tez) since it fails when reduce
is vectorized to get HIVE-7029 checked in.
Stack trace:
{code}
Container released by application,
AttemptID:attempt_1406747677386_0003_2_00_00_2 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:382)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
... 6
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51754 --- metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/24472/#comment90343 Why not just use Warehouse.getFileStatusesForSD(tbl.getSd())? It does the same thing. metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/24472/#comment90349 If I undestand correctly, the difference between this method and the one below is FileStatus[]. If so factor out the common code and pass FileStatus[] as parameter. In case of tempTables you can use WareHouse.getFileStatusesFromSD() API to get FileStatus[]. Correct me if I am wrong. ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java https://reviews.apache.org/r/24472/#comment90353 You can reuse the oldCols, newCols List above instead of using iterator. idx in the for loop is unused. ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java https://reviews.apache.org/r/24472/#comment90358 Is there any reason why you are not using FieldSchema's equals() here? ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java https://reviews.apache.org/r/24472/#comment90331 With my comment below (about using fully qualified column names in place of nested map), this function should become substantially small. It will be like.. ss.getTempTableColStats().get(fullyQualifiedColName) ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java https://reviews.apache.org/r/24472/#comment90334 Same can be done here as well. You can get the fully qualfied col name from colStats object which can be used to update state. - Prasanth_J On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
Re: Review Request 15449: session/operation timeout for hiveserver2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15449/#review51760 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90359 Why did you remove the possible options? common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90357 Why did you remove the possible values? common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90356 Why did you remove the possible options? common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90354 Nit: use camel caps on HiveServer2. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90352 Please restore (in seconds) to description and specify other time units that can be used, if any. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90350 Please restore (in seconds) to description and specify other time units that can be used, if any. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90337 Please add time unit information: Accepts time units like d/h/m/s/ms/us/ns. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90338 Please add time unit information: Accepts time units like d/h/m/s/ms/us/ns. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/15449/#comment90339 Please add time unit information: Accepts time units like d/h/m/s/ms/us/ns. - Lefty Leverenz On Aug. 28, 2014, 2:31 a.m., Navis Ryu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15449/ --- (Updated Aug. 28, 2014, 2:31 a.m.) Review request for hive. Bugs: HIVE-5799 https://issues.apache.org/jira/browse/HIVE-5799 Repository: hive-git Description --- Need some timeout facility for preventing resource leakages from instable or bad clients. Diffs - common/src/java/org/apache/hadoop/hive/ant/GenHiveTemplate.java 4293b7c common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 74bb863 common/src/java/org/apache/hadoop/hive/conf/Validator.java cea9c41 itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2SessionTimeout.java PRE-CREATION service/src/java/org/apache/hive/service/cli/CLIService.java ff5de4a service/src/java/org/apache/hive/service/cli/OperationState.java 3e15f0c service/src/java/org/apache/hive/service/cli/operation/Operation.java 0d6436e service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2867301 service/src/java/org/apache/hive/service/cli/session/HiveSession.java 270e4a6 service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 84e1c7e service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 4e5f595 service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 39d2184 service/src/java/org/apache/hive/service/cli/session/SessionManager.java 17c1c7b service/src/test/org/apache/hive/service/cli/CLIServiceTest.java d01e819 Diff: https://reviews.apache.org/r/15449/diff/ Testing --- Confirmed in the local environment. Thanks, Navis Ryu
[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113514#comment-14113514 ] Lefty Leverenz commented on HIVE-5799: -- [~navis], in HiveConf.java you're removing the possible values from descriptions for *hive.default.fileformat*, *hive.query.result.fileformat*, and *hive.exec.orc.encoding.strategy*. Why is that? Of course HiveConf.java shows the possible values and the wiki can document them, but they won't appear in the generated hive-default.xml.template file. Also, would you please specify units in the descriptions of *hive.server2.session.check.interval*, *hive.server2.idle.session.timeout*, and *hive.server2.idle.operation.timeout*? (Same reason -- for generated hive-default.xml.template.) I posted that request on the review board, as well as the possible values questions. Thanks. session/operation timeout for hiveserver2 - Key: HIVE-5799 URL: https://issues.apache.org/jira/browse/HIVE-5799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5799.1.patch.txt, HIVE-5799.10.patch.txt, HIVE-5799.11.patch.txt, HIVE-5799.12.patch.txt, HIVE-5799.13.patch.txt, HIVE-5799.14.patch.txt, HIVE-5799.15.patch.txt, HIVE-5799.2.patch.txt, HIVE-5799.3.patch.txt, HIVE-5799.4.patch.txt, HIVE-5799.5.patch.txt, HIVE-5799.6.patch.txt, HIVE-5799.7.patch.txt, HIVE-5799.8.patch.txt, HIVE-5799.9.patch.txt Need some timeout facility for preventing resource leakages from instable or bad clients. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113517#comment-14113517
]
Hive QA commented on HIVE-5799:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664805/HIVE-5799.15.patch.txt
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6127 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/540/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/540/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-540/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12664805
session/operation timeout for hiveserver2
-
Key: HIVE-5799
URL: https://issues.apache.org/jira/browse/HIVE-5799
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-5799.1.patch.txt, HIVE-5799.10.patch.txt,
HIVE-5799.11.patch.txt, HIVE-5799.12.patch.txt, HIVE-5799.13.patch.txt,
HIVE-5799.14.patch.txt, HIVE-5799.15.patch.txt, HIVE-5799.2.patch.txt,
HIVE-5799.3.patch.txt, HIVE-5799.4.patch.txt, HIVE-5799.5.patch.txt,
HIVE-5799.6.patch.txt, HIVE-5799.7.patch.txt, HIVE-5799.8.patch.txt,
HIVE-5799.9.patch.txt
Need some timeout facility for preventing resource leakages from instable or
bad clients.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7649) Support column stats with temporary tables
[ https://issues.apache.org/jira/browse/HIVE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113518#comment-14113518 ] Gunther Hagleitner commented on HIVE-7649: -- [~gopalv] do you want to take a look also? Support column stats with temporary tables -- Key: HIVE-7649 URL: https://issues.apache.org/jira/browse/HIVE-7649 Project: Hive Issue Type: Bug Components: Statistics Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-7649.1.patch, HIVE-7649.2.patch, HIVE-7649.3.patch Column stats currently not supported with temp tables, see if they can be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51762 --- ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java https://reviews.apache.org/r/24472/#comment90363 This doesn't look good. Can you use MapString, ColumnStatisticsObj instead? with key being fully qualified column name. StatsUtils.getFullyQualifiedColumnName(String dbname, String tablename, String colname) can be used to generate key. - Prasanth_J On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51763 --- ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q https://reviews.apache.org/r/24472/#comment90365 Can you also add a testcase for partitioned table? similar to columnstats_partlvl.q - Prasanth_J On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
[jira] [Commented] (HIVE-7649) Support column stats with temporary tables
[ https://issues.apache.org/jira/browse/HIVE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113525#comment-14113525 ] Prasanth J commented on HIVE-7649: -- Left some comments on rb Support column stats with temporary tables -- Key: HIVE-7649 URL: https://issues.apache.org/jira/browse/HIVE-7649 Project: Hive Issue Type: Bug Components: Statistics Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-7649.1.patch, HIVE-7649.2.patch, HIVE-7649.3.patch Column stats currently not supported with temp tables, see if they can be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4629) HS2 should support an API to retrieve query logs
[ https://issues.apache.org/jira/browse/HIVE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113540#comment-14113540 ] Lefty Leverenz commented on HIVE-4629: -- Does this need to be documented in the wiki? Of course, two new parameters need to go in Configuration Properties (*hive.server2.logging.operation.enabled*, *hive.server2.logging.operation.log.location*), but should general guidance be given in one of the HS2 wikidocs? The logging section in Getting Started could also mention this. * [HiveServer2 Clients | https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients] * [Setting Up HiveServer2 | https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] * [Getting Started -- Error Logs | https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs] * [Configuration Properties -- HiveServer2 | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2] HS2 should support an API to retrieve query logs Key: HIVE-4629 URL: https://issues.apache.org/jira/browse/HIVE-4629 Project: Hive Issue Type: Sub-task Components: HiveServer2 Reporter: Shreepadma Venugopalan Assignee: Dong Chen Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-4629-no_thrift.1.patch, HIVE-4629.1.patch, HIVE-4629.2.patch, HIVE-4629.3.patch.txt, HIVE-4629.4.patch, HIVE-4629.5.patch, HIVE-4629.6.patch, HIVE-4629.7.patch, HIVE-4629.8.patch, HIVE-4629.9.patch HiveServer2 should support an API to retrieve query logs. This is particularly relevant because HiveServer2 supports async execution but doesn't provide a way to report progress. Providing an API to retrieve query logs will help report progress to the client. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4629) HS2 should support an API to retrieve query logs
[ https://issues.apache.org/jira/browse/HIVE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-4629: - Labels: TODOC14 (was: ) HS2 should support an API to retrieve query logs Key: HIVE-4629 URL: https://issues.apache.org/jira/browse/HIVE-4629 Project: Hive Issue Type: Sub-task Components: HiveServer2 Reporter: Shreepadma Venugopalan Assignee: Dong Chen Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-4629-no_thrift.1.patch, HIVE-4629.1.patch, HIVE-4629.2.patch, HIVE-4629.3.patch.txt, HIVE-4629.4.patch, HIVE-4629.5.patch, HIVE-4629.6.patch, HIVE-4629.7.patch, HIVE-4629.8.patch, HIVE-4629.9.patch HiveServer2 should support an API to retrieve query logs. This is particularly relevant because HiveServer2 supports async execution but doesn't provide a way to report progress. Providing an API to retrieve query logs will help report progress to the client. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7898) HCatStorer should ignore namespaces generated by Pig
[
https://issues.apache.org/jira/browse/HIVE-7898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113576#comment-14113576
]
Hive QA commented on HIVE-7898:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664820/HIVE-7898.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6132 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/541/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/541/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-541/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12664820
HCatStorer should ignore namespaces generated by Pig
Key: HIVE-7898
URL: https://issues.apache.org/jira/browse/HIVE-7898
Project: Hive
Issue Type: Improvement
Components: HCatalog
Affects Versions: 0.13.1
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
Attachments: HIVE-7898.1.patch
Currently, Pig aliases must exactly match the names of HCat columns for
HCatStorer to be successful. However, several Pig operations prepend a
namespace to the alias in order to differentiate fields (e.g. after a group
with field b, you might have A::b). In this case, even if the fields are in
the right order and the alias without namespace matches, the store will fail
because it tries to match the long form of the alias, despite the namespace
being extraneous information in this case. Note that multiple aliases can
be applied (e.g. A::B::C::d).
A workaround is possible by doing a
FOREACH relation GENERATE field1 AS field1, field2 AS field2, etc.
This quickly becomes tedious and bloated for tables with many fields.
Changing this would normally require care around columns named, for example,
`A::b` as has been introduced in Hive 13. However, a different function call
only validates Pig aliases if they follow the old rules for Hive columns. As
such, a direct change (rather than attempting to match either the
namespace::alias or just alias) maintains compatibility for now.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[
https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113634#comment-14113634
]
Hive QA commented on HIVE-6245:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664846/HIVE-6245.5.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6127 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/542/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/542/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-542/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12664846
HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
---
Key: HIVE-6245
URL: https://issues.apache.org/jira/browse/HIVE-6245
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Reporter: Chaoyu Tang
Assignee: Venki Korukanti
Attachments: HIVE-6245.2.patch.txt, HIVE-6245.3.patch.txt,
HIVE-6245.4.patch, HIVE-6245.5.patch, HIVE-6245.patch
The case with following settings is valid but does not work correctly in
current HS2:
==
hive.server2.authentication=NONE (or LDAP)
hive.server2.enable.doAs= true
hive.metastore.sasl.enabled=false
hive.metastore.execute.setugi=true
==
Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC
application) and create DBs/Tables with user's ownership.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7898) HCatStorer should ignore namespaces generated by Pig
[ https://issues.apache.org/jira/browse/HIVE-7898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113716#comment-14113716 ] Justin Leet commented on HIVE-7898: --- These two tests both appear to fail locally with or without my changes. They're also both well outside what hcatalog-pig-adapter, and in specific HCatBaseStorer, would be involved in. Both running query files, and not Pig scripts that use the storer. HCatStorer should ignore namespaces generated by Pig Key: HIVE-7898 URL: https://issues.apache.org/jira/browse/HIVE-7898 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.13.1 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Attachments: HIVE-7898.1.patch Currently, Pig aliases must exactly match the names of HCat columns for HCatStorer to be successful. However, several Pig operations prepend a namespace to the alias in order to differentiate fields (e.g. after a group with field b, you might have A::b). In this case, even if the fields are in the right order and the alias without namespace matches, the store will fail because it tries to match the long form of the alias, despite the namespace being extraneous information in this case. Note that multiple aliases can be applied (e.g. A::B::C::d). A workaround is possible by doing a FOREACH relation GENERATE field1 AS field1, field2 AS field2, etc. This quickly becomes tedious and bloated for tables with many fields. Changing this would normally require care around columns named, for example, `A::b` as has been introduced in Hive 13. However, a different function call only validates Pig aliases if they follow the old rules for Hive columns. As such, a direct change (rather than attempting to match either the namespace::alias or just alias) maintains compatibility for now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7899) txnMgr should be session specific
Alan Gates created HIVE-7899: Summary: txnMgr should be session specific Key: HIVE-7899 URL: https://issues.apache.org/jira/browse/HIVE-7899 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.13.1 Reporter: Alan Gates Assignee: Alan Gates Currently the Driver class keeps a single reference to a transaction manager. It should instead be session specific since in the future multiple queries will want to share the same transaction manager. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7896) orcfiledump should be able to dump data
[ https://issues.apache.org/jira/browse/HIVE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113865#comment-14113865 ] Alan Gates commented on HIVE-7896: -- I don't think my changes are related to this failure. orcfiledump should be able to dump data --- Key: HIVE-7896 URL: https://issues.apache.org/jira/browse/HIVE-7896 Project: Hive Issue Type: Improvement Components: File Formats Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7896.patch The FileDumper utility in orc, exposed as a service as orcfiledump, can print out metadata from Orc files but not the actual data. Being able to dump the data is also useful in some debugging contexts. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 25047: further improve the columns stats update speed for all the partitions of a table
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25047/#review51782 --- metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/25047/#comment90383 You want to persist the object regardless whether it exists or not. Its update (if found) or insert scenario (if not found). So, Why do you need to retrieve MPartitionColumnStatistics? metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/25047/#comment90385 We want to update stats object in such case too. Will that be the case here ? metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/25047/#comment90386 Nested transactions are never a good idea. You are already in a txn, no need to create another one. - Ashutosh Chauhan On Aug. 27, 2014, 10:56 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25047/ --- (Updated Aug. 27, 2014, 10:56 p.m.) Review request for hive. Repository: hive-git Description --- The previous solution https://issues.apache.org/jira/browse/HIVE-7736 is not enough for the case when there are too many columns/partitions. The user will encounter org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out We try to remove more of transaction overhead Diffs - metastore/bin/.gitignore 0dd9890 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 9e3481a metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java e8f16a5 metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e435d69 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 3847d99 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 981fa1a ql/.gitignore 916e17c Diff: https://reviews.apache.org/r/25047/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Updated] (HIVE-7876) further improve the columns stats update speed for all the partitions of a table
[ https://issues.apache.org/jira/browse/HIVE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7876: --- Status: Open (was: Patch Available) Left comments on RB. I don't get why you need to retrieve the object before persisting it. You want to persist regardless whether it exists or not in DB. further improve the columns stats update speed for all the partitions of a table Key: HIVE-7876 URL: https://issues.apache.org/jira/browse/HIVE-7876 Project: Hive Issue Type: Improvement Reporter: pengcheng xiong Assignee: pengcheng xiong Priority: Minor Attachments: HIVE-7876.2.patch, HIVE-7876.3.patch, HIVE-7876.4.patch, HIVE-7876.4.patch The previous solution https://issues.apache.org/jira/browse/HIVE-7736 is not enough for the case when there are too many columns/partitions. The user will encounter org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out We try to remove more of transaction overhead -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7899) txnMgr should be session specific
[ https://issues.apache.org/jira/browse/HIVE-7899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7899: - Attachment: HIVE-7899.patch Attaching a patch that moves the transaction manager from Driver to SessionState. This will make it easier to share the transaction manager across queries in the same session. txnMgr should be session specific - Key: HIVE-7899 URL: https://issues.apache.org/jira/browse/HIVE-7899 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.13.1 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7899.patch Currently the Driver class keeps a single reference to a transaction manager. It should instead be session specific since in the future multiple queries will want to share the same transaction manager. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[ https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113881#comment-14113881 ] Venki Korukanti commented on HIVE-6245: --- Ran 2 failed tests locally and both pass successfully. HS2 creates DBs/Tables with wrong ownership when HMS setugi is true --- Key: HIVE-6245 URL: https://issues.apache.org/jira/browse/HIVE-6245 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0, 0.13.0 Reporter: Chaoyu Tang Assignee: Venki Korukanti Attachments: HIVE-6245.2.patch.txt, HIVE-6245.3.patch.txt, HIVE-6245.4.patch, HIVE-6245.5.patch, HIVE-6245.patch The case with following settings is valid but does not work correctly in current HS2: == hive.server2.authentication=NONE (or LDAP) hive.server2.enable.doAs= true hive.metastore.sasl.enabled=false hive.metastore.execute.setugi=true == Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC application) and create DBs/Tables with user's ownership. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[ https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113901#comment-14113901 ] Alexander Pivovarov commented on HIVE-6245: --- testCliDriver_dynpart_sort_opt_vectorization failed in several previous builds as well. (538-541) TestHiveServer2.testConnection failed in build 540 too http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/540/testReport/ Your build 542 does not have any additional failed test methods. HS2 creates DBs/Tables with wrong ownership when HMS setugi is true --- Key: HIVE-6245 URL: https://issues.apache.org/jira/browse/HIVE-6245 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0, 0.13.0 Reporter: Chaoyu Tang Assignee: Venki Korukanti Attachments: HIVE-6245.2.patch.txt, HIVE-6245.3.patch.txt, HIVE-6245.4.patch, HIVE-6245.5.patch, HIVE-6245.patch The case with following settings is valid but does not work correctly in current HS2: == hive.server2.authentication=NONE (or LDAP) hive.server2.enable.doAs= true hive.metastore.sasl.enabled=false hive.metastore.execute.setugi=true == Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC application) and create DBs/Tables with user's ownership. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[ https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113916#comment-14113916 ] Venki Korukanti commented on HIVE-6245: --- Actually testCliDriver_dynpart_sort_opt_vectorization still fails locally, for some reason it didn't run previously might be due to incorrect maven args in my test run. HS2 creates DBs/Tables with wrong ownership when HMS setugi is true --- Key: HIVE-6245 URL: https://issues.apache.org/jira/browse/HIVE-6245 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0, 0.13.0 Reporter: Chaoyu Tang Assignee: Venki Korukanti Attachments: HIVE-6245.2.patch.txt, HIVE-6245.3.patch.txt, HIVE-6245.4.patch, HIVE-6245.5.patch, HIVE-6245.patch The case with following settings is valid but does not work correctly in current HS2: == hive.server2.authentication=NONE (or LDAP) hive.server2.enable.doAs= true hive.metastore.sasl.enabled=false hive.metastore.execute.setugi=true == Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC application) and create DBs/Tables with user's ownership. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24986: HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24986/ --- (Updated Aug. 28, 2014, 4:23 p.m.) Review request for hive. Bugs: HIVE-7553 https://issues.apache.org/jira/browse/HIVE-7553 Repository: hive-git Description --- HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9d64aff18329e7850342855aade42e21f5 hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 93a03adeab7ba3c3c91344955d303e4252005239 hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClient.java f25039dcf55b3b24bbf8dcba05855665a1c7f3b0 ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java 5924bcf1f55dc4c2dd06f312f929047b7df9de55 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 0c6a3d44ef1f796778768421dc02f8bf3ede6a8c ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java bd45df1a401d1adb009e953d08205c7d5c2d5de2 ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java dcc19f70644c561e17df8c8660ca62805465f1d6 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 76fee612a583cdc2c632d27932623521b735e768 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb2851b2c5fa52e0f555b3d8d1beea5d1ac3b225 ql/src/java/org/apache/hadoop/hive/ql/hooks/HookUtils.java 3f474f846c7af5f1f65f1c14f3ce51308f1279d4 ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 0962cadce0d515e046371d0a816f4efd70b8eef7 ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java 9051ba6d80e619ddbb6c27bb161e1e7a5cdb08a5 ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java edec1b734fb2f015902fd5e1c8afd5acdf4cb3bf ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2e30195a25844a25e9ec8a7c42ed99b75c ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java b15aedc15d8cd0979aced6ff4c9e87606576f0a3 ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java d86df453cd7686627940ade62c0fd72f1636dd0b ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 0a1c660b4bbd46d8410e646270b23c99a4de8b7e ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b05d3b48ec014e4dc8026bb5f6615f62da0e2210 ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 17eeae1a3435fceb4b57325675c58b599e0973ea ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 930acbc98e81f8d421cee1170659d8b7a427fe7d ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 39f1793aaa5bed8a494883cac516ad314be951f4 ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 0d237f01a248a65b4092eb7202fe30eebf27be82 ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java f5bc427a5834860441f21bfc72e175c6a1cf877f ql/src/java/org/apache/hadoop/hive/ql/processors/RefreshProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9798cf3f537a27d1f828f8139790c62c5945c366 ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java e247184b7d95c85fd3e12432e7eb75eb1e2a0b68 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 959007a54b335bb0bdef0256f60e6cbc65798dc7 ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 7fd6c172d18e0d68dc0568848a5fad255a0a9188 ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java ef0052f5763922d50986f127c416af5eaa6ae30d ql/src/test/resources/SessionStateTest.jar.v1 PRE-CREATION ql/src/test/resources/SessionStateTest.jar.v2 PRE-CREATION service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java bc0a02c1df7f9fdb848d5f078e94a663a579e571 Diff: https://reviews.apache.org/r/24986/diff/ Testing --- Thanks, cheng xu
[jira] [Updated] (HIVE-7553) avoid the scheduling maintenance window for every jar change
[ https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-7553: --- Attachment: HIVE-7553.patch add one more case for newly added codes and fix UT issues avoid the scheduling maintenance window for every jar change Key: HIVE-7553 URL: https://issues.apache.org/jira/browse/HIVE-7553 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-7553.patch, HIVE-7553.pdf When user needs to refresh existing or add a new jar to HS2, it needs to restart it. As HS2 is service exposed to clients, this requires scheduling maintenance window for every jar change. It would be great if we could avoid that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7553) avoid the scheduling maintenance window for every jar change
[ https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-7553: --- Attachment: (was: HIVE-7553.patch) avoid the scheduling maintenance window for every jar change Key: HIVE-7553 URL: https://issues.apache.org/jira/browse/HIVE-7553 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-7553.patch, HIVE-7553.pdf When user needs to refresh existing or add a new jar to HS2, it needs to restart it. As HS2 is service exposed to clients, this requires scheduling maintenance window for every jar change. It would be great if we could avoid that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7553) avoid the scheduling maintenance window for every jar change
[ https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-7553: --- Attachment: HIVE-7553.1.patch avoid the scheduling maintenance window for every jar change Key: HIVE-7553 URL: https://issues.apache.org/jira/browse/HIVE-7553 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-7553.1.patch, HIVE-7553.patch, HIVE-7553.pdf When user needs to refresh existing or add a new jar to HS2, it needs to restart it. As HS2 is service exposed to clients, this requires scheduling maintenance window for every jar change. It would be great if we could avoid that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[ https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113959#comment-14113959 ] Thejas M Nair commented on HIVE-6245: - +1 HS2 creates DBs/Tables with wrong ownership when HMS setugi is true --- Key: HIVE-6245 URL: https://issues.apache.org/jira/browse/HIVE-6245 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0, 0.13.0 Reporter: Chaoyu Tang Assignee: Venki Korukanti Attachments: HIVE-6245.2.patch.txt, HIVE-6245.3.patch.txt, HIVE-6245.4.patch, HIVE-6245.5.patch, HIVE-6245.patch The case with following settings is valid but does not work correctly in current HS2: == hive.server2.authentication=NONE (or LDAP) hive.server2.enable.doAs= true hive.metastore.sasl.enabled=false hive.metastore.execute.setugi=true == Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC application) and create DBs/Tables with user's ownership. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change
[
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113962#comment-14113962
]
Hive QA commented on HIVE-7553:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665004/HIVE-7553.1.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/544/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/544/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-544/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-544/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target
itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-unit/target itests/custom-serde/target itests/util/target
hcatalog/target hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
accumulo-handler/target hwi/target common/target common/src/gen service/target
contrib/target serde/target beeline/target odbc/target cli/target
ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1621165.
At revision 1621165.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12665004
avoid the scheduling maintenance window for every jar change
Key: HIVE-7553
URL: https://issues.apache.org/jira/browse/HIVE-7553
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Attachments: HIVE-7553.1.patch, HIVE-7553.patch, HIVE-7553.pdf
When user needs to refresh existing or add a new jar to HS2, it needs to
restart it. As HS2 is service exposed to clients, this requires scheduling
maintenance window for every jar change. It would be great if we could avoid
that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7557) When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
[
https://issues.apache.org/jira/browse/HIVE-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113989#comment-14113989
]
Szehon Ho commented on HIVE-7557:
-
There is some strange error with the build machine not being able to post
comment.. posting below manually:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664847/HIVE-7557.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6126 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/543/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/543/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-543/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12664847
2014-08-28 07:36:44,264 ERROR JIRAService.postComment:165 Encountered error
attempting to post comment to HIVE-7557 java.lang.RuntimeException: 200 OK
at
org.apache.hive.ptest.execution.JIRAService.postComment(JIRAService.java:160)
at
org.apache.hive.ptest.execution.PTest.publishJiraComment(PTest.java:237)
at org.apache.hive.ptest.execution.PTest.run(PTest.java:211)
at
org.apache.hive.ptest.api.server.TestExecutor.run(TestExecutor.java:120)
When reduce is vectorized, dynpart_sort_opt_vectorization.q under Tez fails
---
Key: HIVE-7557
URL: https://issues.apache.org/jira/browse/HIVE-7557
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline
Attachments: HIVE-7557.1.patch
Turned off dynpart_sort_opt_vectorization.q (Tez) since it fails when reduce
is vectorized to get HIVE-7029 checked in.
Stack trace:
{code}
Container released by application,
AttemptID:attempt_1406747677386_0003_2_00_00_2 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be cast to
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:168)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:159)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processVectors(ReduceRecordProcessor.java:481)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processRows(ReduceRecordProcessor.java:371)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing vector batch (tag=0) [Error getting row data with exception
java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector cannot be
Re: Review Request 25047: further improve the columns stats update speed for all the partitions of a table
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25047/ --- (Updated Aug. 28, 2014, 5:39 p.m.) Review request for hive. Changes --- address ashutosh's comments, remove unnecessary txns Repository: hive-git Description --- The previous solution https://issues.apache.org/jira/browse/HIVE-7736 is not enough for the case when there are too many columns/partitions. The user will encounter org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out We try to remove more of transaction overhead Diffs (updated) - metastore/bin/.gitignore 0dd9890 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 9e3481a metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java e8f16a5 metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e435d69 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 3847d99 metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 981fa1a ql/.gitignore 916e17c Diff: https://reviews.apache.org/r/25047/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Updated] (HIVE-7876) further improve the columns stats update speed for all the partitions of a table
[ https://issues.apache.org/jira/browse/HIVE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengcheng xiong updated HIVE-7876: -- Status: Patch Available (was: Open) address ashutosh's comments, remove unnecessary txns further improve the columns stats update speed for all the partitions of a table Key: HIVE-7876 URL: https://issues.apache.org/jira/browse/HIVE-7876 Project: Hive Issue Type: Improvement Reporter: pengcheng xiong Assignee: pengcheng xiong Priority: Minor Attachments: HIVE-7876.2.patch, HIVE-7876.3.patch, HIVE-7876.4.patch, HIVE-7876.4.patch, HIVE-7876.5.patch The previous solution https://issues.apache.org/jira/browse/HIVE-7736 is not enough for the case when there are too many columns/partitions. The user will encounter org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out We try to remove more of transaction overhead -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7876) further improve the columns stats update speed for all the partitions of a table
[ https://issues.apache.org/jira/browse/HIVE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengcheng xiong updated HIVE-7876: -- Attachment: HIVE-7876.5.patch address ashutosh's comments, remove unnecessary txns further improve the columns stats update speed for all the partitions of a table Key: HIVE-7876 URL: https://issues.apache.org/jira/browse/HIVE-7876 Project: Hive Issue Type: Improvement Reporter: pengcheng xiong Assignee: pengcheng xiong Priority: Minor Attachments: HIVE-7876.2.patch, HIVE-7876.3.patch, HIVE-7876.4.patch, HIVE-7876.4.patch, HIVE-7876.5.patch The previous solution https://issues.apache.org/jira/browse/HIVE-7736 is not enough for the case when there are too many columns/partitions. The user will encounter org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out We try to remove more of transaction overhead -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Condit updated HIVE-4605: --- Affects Version/s: 0.12.0 0.13.0 0.13.1 Hive job fails while closing reducer output - Unable to rename -- Key: HIVE-4605 URL: https://issues.apache.org/jira/browse/HIVE-4605 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1 Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Hadoop 1.1.2 Reporter: Link Qian Assignee: Brock Noland Attachments: HIVE-4605.patch 1, create a table with ORC storage model create table iparea_analysis_orc (network int, ip string, ) stored as ORC; 2, insert table iparea_analysis_orc select network, ip, , the script success, but failed after add *OVERWRITE* keyword. the main error log list as here. ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) ... 7 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Condit updated HIVE-4605: --- Attachment: HIVE-4605.patch Hive job fails while closing reducer output - Unable to rename -- Key: HIVE-4605 URL: https://issues.apache.org/jira/browse/HIVE-4605 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1 Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Hadoop 1.1.2 Reporter: Link Qian Assignee: Brock Noland Attachments: HIVE-4605.patch 1, create a table with ORC storage model create table iparea_analysis_orc (network int, ip string, ) stored as ORC; 2, insert table iparea_analysis_orc select network, ip, , the script success, but failed after add *OVERWRITE* keyword. the main error log list as here. ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) ... 7 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Condit reopened HIVE-4605: We are seeing this issue as well. Hive job fails while closing reducer output - Unable to rename -- Key: HIVE-4605 URL: https://issues.apache.org/jira/browse/HIVE-4605 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1 Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Hadoop 1.1.2 Reporter: Link Qian Assignee: Brock Noland Attachments: HIVE-4605.patch 1, create a table with ORC storage model create table iparea_analysis_orc (network int, ip string, ) stored as ORC; 2, insert table iparea_analysis_orc select network, ip, , the script success, but failed after add *OVERWRITE* keyword. the main error log list as here. ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) ... 7 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114044#comment-14114044 ] Craig Condit commented on HIVE-4605: The patch by [~GWong] seems to work for us. Hive job fails while closing reducer output - Unable to rename -- Key: HIVE-4605 URL: https://issues.apache.org/jira/browse/HIVE-4605 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1 Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Hadoop 1.1.2 Reporter: Link Qian Assignee: Brock Noland Attachments: HIVE-4605.patch 1, create a table with ORC storage model create table iparea_analysis_orc (network int, ip string, ) stored as ORC; 2, insert table iparea_analysis_orc select network, ip, , the script success, but failed after add *OVERWRITE* keyword. the main error log list as here. ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) ... 7 more -- This message was sent by Atlassian JIRA (v6.2#6252)
Hive Contributor request
Hi, Please add me to Hive contributor list Jira User name : suma.shivaprasad Thanks Suma
[jira] [Created] (HIVE-7900) Provide SAX Parser based Hive XMLSerde for underlying XML data
Vijaykumar Singh created HIVE-7900: -- Summary: Provide SAX Parser based Hive XMLSerde for underlying XML data Key: HIVE-7900 URL: https://issues.apache.org/jira/browse/HIVE-7900 Project: Hive Issue Type: Wish Components: File Formats Affects Versions: 0.14.0 Reporter: Vijaykumar Singh Priority: Minor Currently, there is not streaming XMLSerde Option available for Hive tables based on XML Scheam Document. This JIRA attempts to Create Tabular Representation of XML data over hive using proposed Serde. This will benefit adhoc XML analysis using external Hive tables. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7896) orcfiledump should be able to dump data
[
https://issues.apache.org/jira/browse/HIVE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114100#comment-14114100
]
Prasanth J commented on HIVE-7896:
--
Pretty useful patch! I took a quick look at the patch. Mostly looks good.
Couple of comments
1) The arg name for rowindex is actually misleading comma separated list of
row indices. It is not CSV of row indices but CSV of column ids for which row
index should be printed. We should either rename the arg or arg name.
2) I tried printing data for the attached ORC file. It fails with the following
exception
{code}
Exception in thread main org.codehaus.jettison.json.JSONException: Value out
of sequence.
at org.codehaus.jettison.json.JSONWriter.append(JSONWriter.java:136)
at org.codehaus.jettison.json.JSONWriter.value(JSONWriter.java:323)
at org.apache.hadoop.hive.ql.io.orc.FileDump.printMap(FileDump.java:250)
at
org.apache.hadoop.hive.ql.io.orc.FileDump.printObject(FileDump.java:311)
at
org.apache.hadoop.hive.ql.io.orc.FileDump.printStruct(FileDump.java:287)
at
org.apache.hadoop.hive.ql.io.orc.FileDump.printObject(FileDump.java:302)
at
org.apache.hadoop.hive.ql.io.orc.FileDump.printJsonData(FileDump.java:352)
at org.apache.hadoop.hive.ql.io.orc.FileDump.printData(FileDump.java:88)
at org.apache.hadoop.hive.ql.io.orc.FileDump.main(FileDump.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{code}
I am attaching the ORC file and corresponding text file for reference.
orcfiledump should be able to dump data
---
Key: HIVE-7896
URL: https://issues.apache.org/jira/browse/HIVE-7896
Project: Hive
Issue Type: Improvement
Components: File Formats
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-7896.patch
The FileDumper utility in orc, exposed as a service as orcfiledump, can print
out metadata from Orc files but not the actual data. Being able to dump the
data is also useful in some debugging contexts.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7896) orcfiledump should be able to dump data
[ https://issues.apache.org/jira/browse/HIVE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-7896: - Attachment: alltypes.orc alltypes2.txt Attaching files orcfiledump should be able to dump data --- Key: HIVE-7896 URL: https://issues.apache.org/jira/browse/HIVE-7896 Project: Hive Issue Type: Improvement Components: File Formats Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7896.patch, alltypes.orc, alltypes2.txt The FileDumper utility in orc, exposed as a service as orcfiledump, can print out metadata from Orc files but not the actual data. Being able to dump the data is also useful in some debugging contexts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7497) Fix some default values in HiveConf
[
https://issues.apache.org/jira/browse/HIVE-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114116#comment-14114116
]
Vaibhav Gumashta commented on HIVE-7497:
[~dongc] I still have one issue with the patch (sorry for coming out so late on
this). We should change HIVE_GLOBAL_INIT_FILE_LOCATION to
HIVE_SERVER2_GLOBAL_INIT_FILE_LOCATION. This will be in sync with the
convention we've followed so far.
Fix some default values in HiveConf
---
Key: HIVE-7497
URL: https://issues.apache.org/jira/browse/HIVE-7497
Project: Hive
Issue Type: Task
Reporter: Brock Noland
Assignee: Dong Chen
Labels: TODOC14
Fix For: 0.14.0
Attachments: HIVE-7497.1.patch, HIVE-7497.patch
HIVE-5160 resolves an env variable at runtime via calling System.getenv(). As
long as the variable is not defined when you run the build null is returned
and the path is not placed in the hive-default,template. However if it is
defined it will populate hive-default.template with a path which will be
different based on the user running the build. We should use
$\{system:HIVE_CONF_DIR\} instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7497) Fix some default values in HiveConf
[
https://issues.apache.org/jira/browse/HIVE-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114118#comment-14114118
]
Vaibhav Gumashta commented on HIVE-7497:
[~dongc] If you're ok with it, I can make the change in one of my patches I'm
working on.
Fix some default values in HiveConf
---
Key: HIVE-7497
URL: https://issues.apache.org/jira/browse/HIVE-7497
Project: Hive
Issue Type: Task
Reporter: Brock Noland
Assignee: Dong Chen
Labels: TODOC14
Fix For: 0.14.0
Attachments: HIVE-7497.1.patch, HIVE-7497.patch
HIVE-5160 resolves an env variable at runtime via calling System.getenv(). As
long as the variable is not defined when you run the build null is returned
and the path is not placed in the hive-default,template. However if it is
defined it will populate hive-default.template with a path which will be
different based on the user running the build. We should use
$\{system:HIVE_CONF_DIR\} instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7352) Queries without tables fail under Tez
[
https://issues.apache.org/jira/browse/HIVE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-7352:
---
Status: Patch Available (was: Open)
+1
Queries without tables fail under Tez
-
Key: HIVE-7352
URL: https://issues.apache.org/jira/browse/HIVE-7352
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.13.1, 0.13.0
Reporter: Craig Condit
Assignee: Gunther Hagleitner
Attachments: HIVE-7352.1.patch.txt, HIVE-7352.2.patch
Hive 0.13.0 added support for queries that do not reference tables (such as
'SELECT 1'). These queries fail under Tez:
{noformat}
Vertex failed as one or more tasks failed. failedTasks:1]
14/07/07 09:54:42 ERROR tez.TezJobMonitor: Vertex failed, vertexName=Map 1,
vertexId=vertex_1404652697071_4487_1_00, diagnostics=[Task failed,
taskId=task_1404652697071_4487_1_00_00,
diagnostics=[AttemptID:attempt_1404652697071_4487_1_00_00_0 Info:Error:
java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not
create a Path from an empty string
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:174)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.init(TezGroupedSplitsInputFormat.java:113)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:79)
at
org.apache.tez.mapreduce.input.MRInput.setupOldRecordReader(MRInput.java:205)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:362)
at
org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:341)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:99)
at
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:68)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:141)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)
Caused by: java.lang.IllegalArgumentException: Can not create a Path from an
empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
at org.apache.hadoop.fs.Path.init(Path.java:135)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.getPath(HiveInputFormat.java:110)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:228)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:171)
... 14 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7497) Fix some default values in HiveConf
[
https://issues.apache.org/jira/browse/HIVE-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114189#comment-14114189
]
Brock Noland commented on HIVE-7497:
[~vaibhavgumashta] since this is commited any additional work would be done in
a follow on JIRA.
Fix some default values in HiveConf
---
Key: HIVE-7497
URL: https://issues.apache.org/jira/browse/HIVE-7497
Project: Hive
Issue Type: Task
Reporter: Brock Noland
Assignee: Dong Chen
Labels: TODOC14
Fix For: 0.14.0
Attachments: HIVE-7497.1.patch, HIVE-7497.patch
HIVE-5160 resolves an env variable at runtime via calling System.getenv(). As
long as the variable is not defined when you run the build null is returned
and the path is not placed in the hive-default,template. However if it is
defined it will populate hive-default.template with a path which will be
different based on the user running the build. We should use
$\{system:HIVE_CONF_DIR\} instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6123) Implement checkstyle in maven
[
https://issues.apache.org/jira/browse/HIVE-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114206#comment-14114206
]
Lars Francke commented on HIVE-6123:
I'd assume this comes with no obligation at all. The way it's currently
implemented just enables anyone to use the checkstyle plugin manually using
{{mvn checkstyle:checkstyle}} or {{mvn checkstyle:check}}.
I suggest implementing any automatism if wanted in a follow-up JIRA.
Implement checkstyle in maven
-
Key: HIVE-6123
URL: https://issues.apache.org/jira/browse/HIVE-6123
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Lars Francke
Attachments: HIVE-6123.1.patch, HIVE-6123.2.patch
ant had a checkstyle target, we should do something similar for maven
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7730) Extend ReadEntity to add accessed columns from query
[
https://issues.apache.org/jira/browse/HIVE-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-7730:
---
Assignee: Xiaomeng Huang
Extend ReadEntity to add accessed columns from query
Key: HIVE-7730
URL: https://issues.apache.org/jira/browse/HIVE-7730
Project: Hive
Issue Type: Bug
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Fix For: 0.14.0
Attachments: HIVE-7730.001.patch, HIVE-7730.002.patch,
HIVE-7730.003.patch, HIVE-7730.004.patch
-Now what we get from HiveSemanticAnalyzerHookContextImpl is limited. If we
have hook of HiveSemanticAnalyzerHook, we may want to get more things from
hookContext. (e.g. the needed colums from query).-
-So we should get instance of HiveSemanticAnalyzerHookContext from
configuration, extends HiveSemanticAnalyzerHookContext with a new
implementation, overide the HiveSemanticAnalyzerHookContext.update() and put
what you want to the class.-
Hive should store accessed columns to ReadEntity when we set
HIVE_STATS_COLLECT_SCANCOLS(or we can add a confVar) is true.
Then external authorization model can get accessed columns when do
authorization in compile before execute. Maybe we will remove
columnAccessInfo from BaseSemanticAnalyzer, old authorization and
AuthorizationModeV2 can get accessed columns from ReadEntity too.
Here is the quick implement in SemanticAnalyzer.analyzeInternal() below:
{code} boolean isColumnInfoNeedForAuth =
SessionState.get().isAuthorizationModeV2()
HiveConf.getBoolVar(conf,
HiveConf.ConfVars.HIVE_AUTHORIZATION_ENABLED);
if (isColumnInfoNeedForAuth
|| HiveConf.getBoolVar(this.conf,
HiveConf.ConfVars.HIVE_STATS_COLLECT_SCANCOLS) == true) {
ColumnAccessAnalyzer columnAccessAnalyzer = new
ColumnAccessAnalyzer(pCtx);
setColumnAccessInfo(columnAccessAnalyzer.analyzeColumnAccess());
}
compiler.compile(pCtx, rootTasks, inputs, outputs);
// TODO:
// after compile, we can put accessed column list to ReadEntity getting
from columnAccessInfo if HIVE_AUTHORIZATION_ENABLED is set true
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7846) authorization api should support group, not assume case insensitive role names
[ https://issues.apache.org/jira/browse/HIVE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-7846: Status: Patch Available (was: Open) authorization api should support group, not assume case insensitive role names -- Key: HIVE-7846 URL: https://issues.apache.org/jira/browse/HIVE-7846 Project: Hive Issue Type: Bug Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-7846.1.patch The case insensitive behavior of roles should be specific to sql standard authorization. Group type for principal also should be disabled at the sql std authorization layer, instead of disallowing it at the API level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7895) Storage based authorization should consider sticky bit for drop actions
[ https://issues.apache.org/jira/browse/HIVE-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-7895: Status: Patch Available (was: Open) Storage based authorization should consider sticky bit for drop actions --- Key: HIVE-7895 URL: https://issues.apache.org/jira/browse/HIVE-7895 Project: Hive Issue Type: Bug Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-7895.1.patch Storage based authorization provides access control for metadata by giving users permissions on metadata that are equivalent to the permission that user has on corresponding data. However, when checking the permissions to drop a metadata object such as database, table or partition, it does not check if the sticky bit is set on the parent dir of objects corresponding dir in hdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24986: HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase
On Aug. 25, 2014, 4:33 a.m., Lefty Leverenz wrote: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 577-578 https://reviews.apache.org/r/24986/diff/2/?file=668073#file668073line577 And these jars can by - ... can be SerDer - SerDe Thanks for the fixes. Looks good. - Lefty --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24986/#review51359 --- On Aug. 28, 2014, 4:23 p.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24986/ --- (Updated Aug. 28, 2014, 4:23 p.m.) Review request for hive. Bugs: HIVE-7553 https://issues.apache.org/jira/browse/HIVE-7553 Repository: hive-git Description --- HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9d64aff18329e7850342855aade42e21f5 hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 93a03adeab7ba3c3c91344955d303e4252005239 hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClient.java f25039dcf55b3b24bbf8dcba05855665a1c7f3b0 ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java 5924bcf1f55dc4c2dd06f312f929047b7df9de55 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 0c6a3d44ef1f796778768421dc02f8bf3ede6a8c ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java bd45df1a401d1adb009e953d08205c7d5c2d5de2 ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java dcc19f70644c561e17df8c8660ca62805465f1d6 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 76fee612a583cdc2c632d27932623521b735e768 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb2851b2c5fa52e0f555b3d8d1beea5d1ac3b225 ql/src/java/org/apache/hadoop/hive/ql/hooks/HookUtils.java 3f474f846c7af5f1f65f1c14f3ce51308f1279d4 ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 0962cadce0d515e046371d0a816f4efd70b8eef7 ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java 9051ba6d80e619ddbb6c27bb161e1e7a5cdb08a5 ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java edec1b734fb2f015902fd5e1c8afd5acdf4cb3bf ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2e30195a25844a25e9ec8a7c42ed99b75c ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java b15aedc15d8cd0979aced6ff4c9e87606576f0a3 ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java d86df453cd7686627940ade62c0fd72f1636dd0b ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 0a1c660b4bbd46d8410e646270b23c99a4de8b7e ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b05d3b48ec014e4dc8026bb5f6615f62da0e2210 ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 17eeae1a3435fceb4b57325675c58b599e0973ea ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 930acbc98e81f8d421cee1170659d8b7a427fe7d ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 39f1793aaa5bed8a494883cac516ad314be951f4 ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 0d237f01a248a65b4092eb7202fe30eebf27be82 ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java f5bc427a5834860441f21bfc72e175c6a1cf877f ql/src/java/org/apache/hadoop/hive/ql/processors/RefreshProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9798cf3f537a27d1f828f8139790c62c5945c366 ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java e247184b7d95c85fd3e12432e7eb75eb1e2a0b68 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 959007a54b335bb0bdef0256f60e6cbc65798dc7 ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 7fd6c172d18e0d68dc0568848a5fad255a0a9188 ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java ef0052f5763922d50986f127c416af5eaa6ae30d ql/src/test/resources/SessionStateTest.jar.v1 PRE-CREATION ql/src/test/resources/SessionStateTest.jar.v2 PRE-CREATION service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java bc0a02c1df7f9fdb848d5f078e94a663a579e571 Diff: https://reviews.apache.org/r/24986/diff/ Testing --- Thanks, cheng xu
[jira] [Assigned] (HIVE-7870) Insert overwrite table query does not generate correct task plan
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Yang reassigned HIVE-7870:
-
Assignee: Na Yang (was: Chao)
Insert overwrite table query does not generate correct task plan
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Na Yang
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7870) Insert overwrite table query does not generate correct task plan
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114225#comment-14114225
]
Na Yang commented on HIVE-7870:
---
Chao, do you mind that I assign this task back to me?
Insert overwrite table query does not generate correct task plan
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Na Yang
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 24986: HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24986/#review51811 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/24986/#comment90421 Description says renewed by executing reload command but hive.security.command.whitelist values add refresh instead of reload so is this an error here? - Lefty Leverenz On Aug. 28, 2014, 4:23 p.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24986/ --- (Updated Aug. 28, 2014, 4:23 p.m.) Review request for hive. Bugs: HIVE-7553 https://issues.apache.org/jira/browse/HIVE-7553 Repository: hive-git Description --- HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9d64aff18329e7850342855aade42e21f5 hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 93a03adeab7ba3c3c91344955d303e4252005239 hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClient.java f25039dcf55b3b24bbf8dcba05855665a1c7f3b0 ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java 5924bcf1f55dc4c2dd06f312f929047b7df9de55 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 0c6a3d44ef1f796778768421dc02f8bf3ede6a8c ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java bd45df1a401d1adb009e953d08205c7d5c2d5de2 ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java dcc19f70644c561e17df8c8660ca62805465f1d6 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 76fee612a583cdc2c632d27932623521b735e768 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb2851b2c5fa52e0f555b3d8d1beea5d1ac3b225 ql/src/java/org/apache/hadoop/hive/ql/hooks/HookUtils.java 3f474f846c7af5f1f65f1c14f3ce51308f1279d4 ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 0962cadce0d515e046371d0a816f4efd70b8eef7 ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java 9051ba6d80e619ddbb6c27bb161e1e7a5cdb08a5 ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java edec1b734fb2f015902fd5e1c8afd5acdf4cb3bf ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2e30195a25844a25e9ec8a7c42ed99b75c ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java b15aedc15d8cd0979aced6ff4c9e87606576f0a3 ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java d86df453cd7686627940ade62c0fd72f1636dd0b ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 0a1c660b4bbd46d8410e646270b23c99a4de8b7e ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b05d3b48ec014e4dc8026bb5f6615f62da0e2210 ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 17eeae1a3435fceb4b57325675c58b599e0973ea ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 930acbc98e81f8d421cee1170659d8b7a427fe7d ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 39f1793aaa5bed8a494883cac516ad314be951f4 ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 0d237f01a248a65b4092eb7202fe30eebf27be82 ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java f5bc427a5834860441f21bfc72e175c6a1cf877f ql/src/java/org/apache/hadoop/hive/ql/processors/RefreshProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9798cf3f537a27d1f828f8139790c62c5945c366 ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java e247184b7d95c85fd3e12432e7eb75eb1e2a0b68 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 959007a54b335bb0bdef0256f60e6cbc65798dc7 ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 7fd6c172d18e0d68dc0568848a5fad255a0a9188 ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java ef0052f5763922d50986f127c416af5eaa6ae30d ql/src/test/resources/SessionStateTest.jar.v1 PRE-CREATION ql/src/test/resources/SessionStateTest.jar.v2 PRE-CREATION service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java bc0a02c1df7f9fdb848d5f078e94a663a579e571 Diff: https://reviews.apache.org/r/24986/diff/ Testing --- Thanks, cheng xu
[jira] [Updated] (HIVE-7669) parallel order by clause on a string column fails with IOException: Split points are out of order
[
https://issues.apache.org/jira/browse/HIVE-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7669:
Release Note:
Background: HIVE-1402 added parallelism (multiple reducers) to an order-by
query, enabled by hive.optimize.sampling.orderby. As order-by requires total
order, partitioning data among reducers uses a range-partition, and thus
sampling is needed to give us the values for the ranges of each reducer (eg,
1-100, 100-200, etc).
HIVE-7669 fixes the algorithm to generate a better range-partition split when
the sample data lacks enough unique values, and throws exception if the
calculated number of reducers falls below a certain threshold.
Yea, I agree it needs explanation.
I gave a shot at release note just to give my understanding, though I dont know
too much and Navis is the expert. Feel free to take a look and correct as
necessary.
parallel order by clause on a string column fails with IOException: Split
points are out of order
-
Key: HIVE-7669
URL: https://issues.apache.org/jira/browse/HIVE-7669
Project: Hive
Issue Type: Bug
Components: HiveServer2, Query Processor, SQL
Affects Versions: 0.12.0
Environment: Hive 0.12.0-cdh5.0.0
OS: Redhat linux
Reporter: Vishal Kamath
Assignee: Navis
Labels: orderby
Attachments: HIVE-7669.1.patch.txt, HIVE-7669.2.patch.txt,
HIVE-7669.3.patch.txt
The source table has 600 Million rows and it has a String column
l_shipinstruct which has 4 unique values. (Ie. these 4 values are repeated
across the 600 million rows)
We are sorting it based on this string column l_shipinstruct as shown in
the below HiveQL with the following parameters.
{code:sql}
set hive.optimize.sampling.orderby=true;
set hive.optimize.sampling.orderby.number=1000;
set hive.optimize.sampling.orderby.percent=0.1f;
insert overwrite table lineitem_temp_report
select
l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity,
l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate,
l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment
from
lineitem
order by l_shipinstruct;
{code}
Stack Trace
Diagnostic Messages for this Task:
{noformat}
Error: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.init(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 10 more
Caused by: java.lang.IllegalArgumentException: Can't read partitions file
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:42)
at
org.apache.hadoop.hive.ql.exec.HiveTotalOrderPartitioner.configure(HiveTotalOrderPartitioner.java:37)
... 15 more
Caused by: java.io.IOException: Split points are out of order
at
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:96)
... 17 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7870) Insert overwrite table query does not generate correct task plan
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114231#comment-14114231
]
Chao commented on HIVE-7870:
Hi [~nyang], no problem. Just take it.
Insert overwrite table query does not generate correct task plan
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Na Yang
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7497) Fix some default values in HiveConf
[
https://issues.apache.org/jira/browse/HIVE-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114242#comment-14114242
]
Vaibhav Gumashta commented on HIVE-7497:
[~brocknoland] Sure, that's what I intend to do.
Fix some default values in HiveConf
---
Key: HIVE-7497
URL: https://issues.apache.org/jira/browse/HIVE-7497
Project: Hive
Issue Type: Task
Reporter: Brock Noland
Assignee: Dong Chen
Labels: TODOC14
Fix For: 0.14.0
Attachments: HIVE-7497.1.patch, HIVE-7497.patch
HIVE-5160 resolves an env variable at runtime via calling System.getenv(). As
long as the variable is not defined when you run the build null is returned
and the path is not placed in the hive-default,template. However if it is
defined it will populate hive-default.template with a path which will be
different based on the user running the build. We should use
$\{system:HIVE_CONF_DIR\} instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
On Aug. 28, 2014, 8:02 a.m., Prasanth_J wrote: ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q, line 1 https://reviews.apache.org/r/24472/diff/1/?file=655372#file655372line1 Can you also add a testcase for partitioned table? similar to columnstats_partlvl.q Not currently supporting partitioned temp tables. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51763 --- On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
On Aug. 28, 2014, 7:59 a.m., Prasanth_J wrote: ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java, line 89 https://reviews.apache.org/r/24472/diff/1/?file=655370#file655370line89 This doesn't look good. Can you use MapString, ColumnStatisticsObj instead? with key being fully qualified column name. StatsUtils.getFullyQualifiedColumnName(String dbname, String tablename, String colname) can be used to generate key. There are a couple of places in the patch where we want to delete all of the column stats for a table, which gets harder to do if you can only look up the stats based on dbname.tabname.colname. How about I get rid of one level of nested maps by using key tabname.dbname - so MapString, MapString, ColumnStatisticsObj? This would give me an easy way to drop all col stats for one table. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51762 --- On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
On Aug. 28, 2014, 7:56 a.m., Prasanth_J wrote: metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java, line 202 https://reviews.apache.org/r/24472/diff/1/?file=655367#file655367line202 Why not just use Warehouse.getFileStatusesForSD(tbl.getSd())? It does the same thing. True, this does seem to do the same thing. Will use Warehouse.getFileStatusesForSD(), though with your suggestion below this method will disappear. On Aug. 28, 2014, 7:56 a.m., Prasanth_J wrote: metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java, line 239 https://reviews.apache.org/r/24472/diff/1/?file=655367#file655367line239 If I undestand correctly, the difference between this method and the one below is FileStatus[]. If so factor out the common code and pass FileStatus[] as parameter. In case of tempTables you can use WareHouse.getFileStatusesFromSD() API to get FileStatus[]. Correct me if I am wrong. Good, suggestion, I think this should work. On Aug. 28, 2014, 7:56 a.m., Prasanth_J wrote: ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java, line 396 https://reviews.apache.org/r/24472/diff/1/?file=655368#file655368line396 Is there any reason why you are not using FieldSchema's equals() here? FieldSchema.equals() also compares the column comment, which could be changed during alter table. If just the column comment changed the columns are still relatively similar. On Aug. 28, 2014, 7:56 a.m., Prasanth_J wrote: ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java, line 391 https://reviews.apache.org/r/24472/diff/1/?file=655368#file655368line391 You can reuse the oldCols, newCols List above instead of using iterator. idx in the for loop is unused. Thought the iterators would be better depending on what kind of List was used. I can redo the loop without using idx. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51754 --- On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change
[ https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114245#comment-14114245 ] Lefty Leverenz commented on HIVE-7553: -- Doc question (also posted on the review board): one parameter description says renewed by executing reload command but the other adds refresh to its default values. Which is correct, reload or refresh? Answering myself: The first patch had refresh for both, but patch 1 changes hive.refresh.aux.jars.path to hive.reloadable.aux.jars.path so I guess my review board question was backwards -- reload is correct and you need to change refresh to reload in the default values for hive.security.command.whitelist. Otherwise the parameters are good to go. Thanks for the fixes, [~Ferd]. avoid the scheduling maintenance window for every jar change Key: HIVE-7553 URL: https://issues.apache.org/jira/browse/HIVE-7553 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Ferdinand Xu Assignee: Ferdinand Xu Attachments: HIVE-7553.1.patch, HIVE-7553.patch, HIVE-7553.pdf When user needs to refresh existing or add a new jar to HS2, it needs to restart it. As HS2 is service exposed to clients, this requires scheduling maintenance window for every jar change. It would be great if we could avoid that. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51815 --- ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java https://reviews.apache.org/r/24472/#comment90424 This is wrong - deleting all stats on a table when we only want to delete stats for one column. Will fix. - Jason Dere On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
Re: Review Request 25125: HIVE-7895 : Storage based authorization should consider sticky bit for drop actions
On Aug. 28, 2014, 5:13 a.m., Jason Dere wrote: ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java, line 191 https://reviews.apache.org/r/25125/diff/1/?file=670613#file670613line191 Is it necessary to check the delete privs on the parent dir for external tables, given that the table directory is not actually deleted for external tables? Thanks for pointing this out! I forgot about the external table use case! - Thejas --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25125/#review51727 --- On Aug. 27, 2014, 10:30 p.m., Thejas Nair wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25125/ --- (Updated Aug. 27, 2014, 10:30 p.m.) Review request for hive, Jason Dere and Sushanth Sowmyan. Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7895 Diffs - itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestStorageBasedMetastoreAuthorizationDrops.java PRE-CREATION itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestStorageBasedMetastoreAuthorizationProvider.java b447204 ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java ddbe30c Diff: https://reviews.apache.org/r/25125/diff/ Testing --- New tests included. Thanks, Thejas Nair
Re: Review Request 25125: HIVE-7895 : Storage based authorization should consider sticky bit for drop actions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25125/ --- (Updated Aug. 28, 2014, 8:16 p.m.) Review request for hive, Jason Dere and Sushanth Sowmyan. Changes --- addressing review comments Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7895 Diffs (updated) - common/src/java/org/apache/hadoop/hive/common/FileUtils.java f71bc3c itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestStorageBasedMetastoreAuthorizationDrops.java PRE-CREATION itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestStorageBasedMetastoreAuthorizationProvider.java b447204 ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java ddbe30c Diff: https://reviews.apache.org/r/25125/diff/ Testing --- New tests included. Thanks, Thejas Nair
[jira] [Updated] (HIVE-7895) Storage based authorization should consider sticky bit for drop actions
[ https://issues.apache.org/jira/browse/HIVE-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-7895: Attachment: HIVE-7895.2.patch Storage based authorization should consider sticky bit for drop actions --- Key: HIVE-7895 URL: https://issues.apache.org/jira/browse/HIVE-7895 Project: Hive Issue Type: Bug Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-7895.1.patch, HIVE-7895.2.patch Storage based authorization provides access control for metadata by giving users permissions on metadata that are equivalent to the permission that user has on corresponding data. However, when checking the permissions to drop a metadata object such as database, table or partition, it does not check if the sticky bit is set on the parent dir of objects corresponding dir in hdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7870) Insert overwrite table query does not generate correct task plan [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-7870:
--
Summary: Insert overwrite table query does not generate correct task plan
[Spark Branch] (was: Insert overwrite table query does not generate correct
task plan)
Insert overwrite table query does not generate correct task plan [Spark
Branch]
---
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Na Yang
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6633) pig -useHCatalog with embedded metastore fails to pass command line args to metastore
[ https://issues.apache.org/jira/browse/HIVE-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114271#comment-14114271 ] Sushanth Sowmyan commented on HIVE-6633: Hi, I'm afraid this patch modified only org.apache.hcatalog.* files, and those were pruned out with the removal of deprecated classes for 0.14. So this patch was committed to trunk, but the changes effectively blown away. To make commit log tracking easier in this scenario, I'm going to clone this jira to track addition of this same patch to org.apache.hive.hcatalog as well. Eric, could you please regenerate your patch for that? pig -useHCatalog with embedded metastore fails to pass command line args to metastore - Key: HIVE-6633 URL: https://issues.apache.org/jira/browse/HIVE-6633 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.14.0 Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: HIVE-6633.01.patch This fails because the embedded metastore can't connect to the database because the command line -D arguments passed to pig are not getting passed to the metastore when the embedded metastore is created. Using hive.metastore.uris set to the empty string causes creation of an embedded metastore. pig -useHCatalog -Dhive.metastore.uris= -Djavax.jdo.option.ConnectionPassword=AzureSQLDBXYZ The goal is to allow a pig job submitted via WebHCat to specify a metastore to use via job arguments. That is not working because it is not possible to pass Djavax.jdo.option.ConnectionPassword and other necessary arguments to the embedded metastore. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7901) CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version)
Sushanth Sowmyan created HIVE-7901: -- Summary: CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version) Key: HIVE-7901 URL: https://issues.apache.org/jira/browse/HIVE-7901 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.14.0 Reporter: Sushanth Sowmyan Assignee: Eric Hanson Fix For: 0.13.0 This fails because the embedded metastore can't connect to the database because the command line -D arguments passed to pig are not getting passed to the metastore when the embedded metastore is created. Using hive.metastore.uris set to the empty string causes creation of an embedded metastore. pig -useHCatalog -Dhive.metastore.uris= -Djavax.jdo.option.ConnectionPassword=AzureSQLDBXYZ The goal is to allow a pig job submitted via WebHCat to specify a metastore to use via job arguments. That is not working because it is not possible to pass Djavax.jdo.option.ConnectionPassword and other necessary arguments to the embedded metastore. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HIVE-6633) pig -useHCatalog with embedded metastore fails to pass command line args to metastore
[ https://issues.apache.org/jira/browse/HIVE-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan resolved HIVE-6633. Resolution: Fixed Re-marking as resolved, and tracking the org.apache.hive.hcatalog issue on HIVE-7901 pig -useHCatalog with embedded metastore fails to pass command line args to metastore - Key: HIVE-6633 URL: https://issues.apache.org/jira/browse/HIVE-6633 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.14.0 Reporter: Eric Hanson Assignee: Eric Hanson Fix For: 0.13.0 Attachments: HIVE-6633.01.patch This fails because the embedded metastore can't connect to the database because the command line -D arguments passed to pig are not getting passed to the metastore when the embedded metastore is created. Using hive.metastore.uris set to the empty string causes creation of an embedded metastore. pig -useHCatalog -Dhive.metastore.uris= -Djavax.jdo.option.ConnectionPassword=AzureSQLDBXYZ The goal is to allow a pig job submitted via WebHCat to specify a metastore to use via job arguments. That is not working because it is not possible to pass Djavax.jdo.option.ConnectionPassword and other necessary arguments to the embedded metastore. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7901) CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version)
[ https://issues.apache.org/jira/browse/HIVE-7901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-7901: --- Fix Version/s: (was: 0.13.0) CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version) Key: HIVE-7901 URL: https://issues.apache.org/jira/browse/HIVE-7901 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Reporter: Sushanth Sowmyan Assignee: Eric Hanson This fails because the embedded metastore can't connect to the database because the command line -D arguments passed to pig are not getting passed to the metastore when the embedded metastore is created. Using hive.metastore.uris set to the empty string causes creation of an embedded metastore. pig -useHCatalog -Dhive.metastore.uris= -Djavax.jdo.option.ConnectionPassword=AzureSQLDBXYZ The goal is to allow a pig job submitted via WebHCat to specify a metastore to use via job arguments. That is not working because it is not possible to pass Djavax.jdo.option.ConnectionPassword and other necessary arguments to the embedded metastore. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7901) CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version)
[ https://issues.apache.org/jira/browse/HIVE-7901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114275#comment-14114275 ] Sushanth Sowmyan commented on HIVE-7901: This issue is a clone of a prior jira : HIVE-6633, which modified it for org.apache.hcatalog.* classes, which got removed as deprecated classes were pruned out. So, we effectively need to add that change back in, to org.apache.hive.hcatalog.* classes. CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version) Key: HIVE-7901 URL: https://issues.apache.org/jira/browse/HIVE-7901 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Reporter: Sushanth Sowmyan Assignee: Eric Hanson This fails because the embedded metastore can't connect to the database because the command line -D arguments passed to pig are not getting passed to the metastore when the embedded metastore is created. Using hive.metastore.uris set to the empty string causes creation of an embedded metastore. pig -useHCatalog -Dhive.metastore.uris= -Djavax.jdo.option.ConnectionPassword=AzureSQLDBXYZ The goal is to allow a pig job submitted via WebHCat to specify a metastore to use via job arguments. That is not working because it is not possible to pass Djavax.jdo.option.ConnectionPassword and other necessary arguments to the embedded metastore. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7901) CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version)
[ https://issues.apache.org/jira/browse/HIVE-7901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-7901: --- Affects Version/s: (was: 0.13.0) (was: 0.12.0) (was: 0.11.0) CLONE - pig -useHCatalog with embedded metastore fails to pass command line args to metastore (org.apache.hive.hcatalog version) Key: HIVE-7901 URL: https://issues.apache.org/jira/browse/HIVE-7901 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Reporter: Sushanth Sowmyan Assignee: Eric Hanson This fails because the embedded metastore can't connect to the database because the command line -D arguments passed to pig are not getting passed to the metastore when the embedded metastore is created. Using hive.metastore.uris set to the empty string causes creation of an embedded metastore. pig -useHCatalog -Dhive.metastore.uris= -Djavax.jdo.option.ConnectionPassword=AzureSQLDBXYZ The goal is to allow a pig job submitted via WebHCat to specify a metastore to use via job arguments. That is not working because it is not possible to pass Djavax.jdo.option.ConnectionPassword and other necessary arguments to the embedded metastore. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7876) further improve the columns stats update speed for all the partitions of a table
[
https://issues.apache.org/jira/browse/HIVE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114287#comment-14114287
]
Hive QA commented on HIVE-7876:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665025/HIVE-7876.5.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6126 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/545/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/545/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-545/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12665025
further improve the columns stats update speed for all the partitions of a
table
Key: HIVE-7876
URL: https://issues.apache.org/jira/browse/HIVE-7876
Project: Hive
Issue Type: Improvement
Reporter: pengcheng xiong
Assignee: pengcheng xiong
Priority: Minor
Attachments: HIVE-7876.2.patch, HIVE-7876.3.patch, HIVE-7876.4.patch,
HIVE-7876.4.patch, HIVE-7876.5.patch
The previous solution https://issues.apache.org/jira/browse/HIVE-7736
is not enough for the case when there are too many columns/partitions.
The user will encounter
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
We try to remove more of transaction overhead
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.
[ https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114292#comment-14114292 ] david serafini commented on HIVE-7100: -- No. I'm not set up to use RB. Users of hive should be able to specify skipTrash when dropping tables. --- Key: HIVE-7100 URL: https://issues.apache.org/jira/browse/HIVE-7100 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Ravi Prakash Assignee: Jayesh Attachments: HIVE-7100.1.patch, HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.patch Users of our clusters are often running up against their quota limits because of Hive tables. When they drop tables, they have to then manually delete the files from HDFS using skipTrash. This is cumbersome and unnecessary. We should enable users to skipTrash directly when dropping tables. We should also be able to provide this functionality without polluting SQL syntax. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24986: HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24986/#review51822 --- Looks great! As Lefty mentioned we'll need to change the refresh command to reload. ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/24986/#comment90435 Let's put these log messages at debug ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/24986/#comment90436 debug ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java https://reviews.apache.org/r/24986/#comment90437 debug ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java https://reviews.apache.org/r/24986/#comment90438 Can this be final? If so, please make it final. Also we should move it to the top of the class. ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java https://reviews.apache.org/r/24986/#comment90439 Why not call clear and addAll? service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java https://reviews.apache.org/r/24986/#comment90440 this should be error - Brock Noland On Aug. 28, 2014, 4:23 p.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24986/ --- (Updated Aug. 28, 2014, 4:23 p.m.) Review request for hive. Bugs: HIVE-7553 https://issues.apache.org/jira/browse/HIVE-7553 Repository: hive-git Description --- HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9d64aff18329e7850342855aade42e21f5 hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 93a03adeab7ba3c3c91344955d303e4252005239 hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClient.java f25039dcf55b3b24bbf8dcba05855665a1c7f3b0 ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java 5924bcf1f55dc4c2dd06f312f929047b7df9de55 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 0c6a3d44ef1f796778768421dc02f8bf3ede6a8c ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java bd45df1a401d1adb009e953d08205c7d5c2d5de2 ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java dcc19f70644c561e17df8c8660ca62805465f1d6 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 76fee612a583cdc2c632d27932623521b735e768 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb2851b2c5fa52e0f555b3d8d1beea5d1ac3b225 ql/src/java/org/apache/hadoop/hive/ql/hooks/HookUtils.java 3f474f846c7af5f1f65f1c14f3ce51308f1279d4 ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 0962cadce0d515e046371d0a816f4efd70b8eef7 ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java 9051ba6d80e619ddbb6c27bb161e1e7a5cdb08a5 ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java edec1b734fb2f015902fd5e1c8afd5acdf4cb3bf ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2e30195a25844a25e9ec8a7c42ed99b75c ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java b15aedc15d8cd0979aced6ff4c9e87606576f0a3 ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java d86df453cd7686627940ade62c0fd72f1636dd0b ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 0a1c660b4bbd46d8410e646270b23c99a4de8b7e ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b05d3b48ec014e4dc8026bb5f6615f62da0e2210 ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 17eeae1a3435fceb4b57325675c58b599e0973ea ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 930acbc98e81f8d421cee1170659d8b7a427fe7d ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 39f1793aaa5bed8a494883cac516ad314be951f4 ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 0d237f01a248a65b4092eb7202fe30eebf27be82 ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java f5bc427a5834860441f21bfc72e175c6a1cf877f ql/src/java/org/apache/hadoop/hive/ql/processors/RefreshProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9798cf3f537a27d1f828f8139790c62c5945c366 ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java e247184b7d95c85fd3e12432e7eb75eb1e2a0b68 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 959007a54b335bb0bdef0256f60e6cbc65798dc7 ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java 7fd6c172d18e0d68dc0568848a5fad255a0a9188
[jira] [Commented] (HIVE-7876) further improve the columns stats update speed for all the partitions of a table
[ https://issues.apache.org/jira/browse/HIVE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114301#comment-14114301 ] pengcheng xiong commented on HIVE-7876: --- Hi QA, I ran the same test but receive [INFO] BUILD SUCCESS... further improve the columns stats update speed for all the partitions of a table Key: HIVE-7876 URL: https://issues.apache.org/jira/browse/HIVE-7876 Project: Hive Issue Type: Improvement Reporter: pengcheng xiong Assignee: pengcheng xiong Priority: Minor Attachments: HIVE-7876.2.patch, HIVE-7876.3.patch, HIVE-7876.4.patch, HIVE-7876.4.patch, HIVE-7876.5.patch The previous solution https://issues.apache.org/jira/browse/HIVE-7736 is not enough for the case when there are too many columns/partitions. The user will encounter org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out We try to remove more of transaction overhead -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.
[ https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114317#comment-14114317 ] Lefty Leverenz commented on HIVE-7100: -- You can set up a review board account here: https://reviews.apache.org/account/register/. RB documentation: https://www.reviewboard.org/docs/manual/2.0/ Hive review requests: https://reviews.apache.org/groups/hive/ Users of hive should be able to specify skipTrash when dropping tables. --- Key: HIVE-7100 URL: https://issues.apache.org/jira/browse/HIVE-7100 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Ravi Prakash Assignee: Jayesh Attachments: HIVE-7100.1.patch, HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.patch Users of our clusters are often running up against their quota limits because of Hive tables. When they drop tables, they have to then manually delete the files from HDFS using skipTrash. This is cumbersome and unnecessary. We should enable users to skipTrash directly when dropping tables. We should also be able to provide this functionality without polluting SQL syntax. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7482) The execution side changes for SMB join in hive-tez
[ https://issues.apache.org/jira/browse/HIVE-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-7482: - Attachment: HIVE-7482.1.patch Address review comments. The execution side changes for SMB join in hive-tez --- Key: HIVE-7482 URL: https://issues.apache.org/jira/browse/HIVE-7482 Project: Hive Issue Type: Bug Components: Tez Affects Versions: tez-branch Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-7482.1.patch, HIVE-7482.WIP.2.patch, HIVE-7482.WIP.3.patch, HIVE-7482.WIP.4.patch, HIVE-7482.WIP.patch A piece of HIVE-7430. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7876) further improve the columns stats update speed for all the partitions of a table
[ https://issues.apache.org/jira/browse/HIVE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114326#comment-14114326 ] Ashutosh Chauhan commented on HIVE-7876: +1 further improve the columns stats update speed for all the partitions of a table Key: HIVE-7876 URL: https://issues.apache.org/jira/browse/HIVE-7876 Project: Hive Issue Type: Improvement Reporter: pengcheng xiong Assignee: pengcheng xiong Priority: Minor Attachments: HIVE-7876.2.patch, HIVE-7876.3.patch, HIVE-7876.4.patch, HIVE-7876.4.patch, HIVE-7876.5.patch The previous solution https://issues.apache.org/jira/browse/HIVE-7736 is not enough for the case when there are too many columns/partitions. The user will encounter org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out We try to remove more of transaction overhead -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
On Aug. 28, 2014, 7:59 a.m., Prasanth_J wrote: ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java, line 89 https://reviews.apache.org/r/24472/diff/1/?file=655370#file655370line89 This doesn't look good. Can you use MapString, ColumnStatisticsObj instead? with key being fully qualified column name. StatsUtils.getFullyQualifiedColumnName(String dbname, String tablename, String colname) can be used to generate key. Jason Dere wrote: There are a couple of places in the patch where we want to delete all of the column stats for a table, which gets harder to do if you can only look up the stats based on dbname.tabname.colname. How about I get rid of one level of nested maps by using key tabname.dbname - so MapString, MapString, ColumnStatisticsObj? This would give me an easy way to drop all col stats for one table. Getting rid of one level sounds good. - Prasanth_J --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51762 --- On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
On Aug. 28, 2014, 8:02 a.m., Prasanth_J wrote: ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q, line 1 https://reviews.apache.org/r/24472/diff/1/?file=655372#file655372line1 Can you also add a testcase for partitioned table? similar to columnstats_partlvl.q Jason Dere wrote: Not currently supporting partitioned temp tables. Will it throw an exception in that case? If so can you add a NegativeCliDriver test just to make sure it throws some exception if used with partitioned tables. - Prasanth_J --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51763 --- On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
Re: Review Request 24472: HIVE-7649: Support column stats with temporary tables
On Aug. 28, 2014, 7:56 a.m., Prasanth_J wrote: ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java, line 396 https://reviews.apache.org/r/24472/diff/1/?file=655368#file655368line396 Is there any reason why you are not using FieldSchema's equals() here? Jason Dere wrote: FieldSchema.equals() also compares the column comment, which could be changed during alter table. If just the column comment changed the columns are still relatively similar. Can you add equalsIgnoreComment() to FieldSchema then? - Prasanth_J --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/#review51754 --- On Aug. 26, 2014, 6:37 p.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24472/ --- (Updated Aug. 26, 2014, 6:37 p.m.) Review request for hive and Prasanth_J. Bugs: HIVE-7649 https://issues.apache.org/jira/browse/HIVE-7649 Repository: hive-git Description --- Update SessionHiveMetastoreClient to get column stats to work for temp tables. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 5a56ced ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 37b1669 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 24f3710 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java fcfcf42 ql/src/test/queries/clientnegative/temp_table_column_stats.q 9b7aa4a ql/src/test/queries/clientpositive/temp_table_display_colstats_tbllvl.q PRE-CREATION ql/src/test/results/clientnegative/temp_table_column_stats.q.out 4b0c0bc ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24472/diff/ Testing --- Thanks, Jason Dere
[jira] [Created] (HIVE-7902) Cleanup hbase-handler/pom.xml dependency list
Venki Korukanti created HIVE-7902:
-
Summary: Cleanup hbase-handler/pom.xml dependency list
Key: HIVE-7902
URL: https://issues.apache.org/jira/browse/HIVE-7902
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.13.1, 0.13.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Priority: Minor
Fix For: 0.14.0
Noticed an extra dependency {{hive-service}} when changing dependency version
of {{hive-hbase-handler}} from 0.12.0 to 0.13.0 in a third party application.
Tracing the log of hbase-handler/pom.xml file, it is added as part of ant to
maven migration and not because of any specific functionality requirement.
Dependency {{hive-service}} is not needed in {{hive-hbase-handler}} and can be
removed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7881) enable Qtest scriptfile1.q [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-7881: Summary: enable Qtest scriptfile1.q [Spark Branch] (was: enable Qtest scriptfiel1.q[Spark Branch]) enable Qtest scriptfile1.q [Spark Branch] - Key: HIVE-7881 URL: https://issues.apache.org/jira/browse/HIVE-7881 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M1 Attachments: HIVE-7881.1-spark.patch scriptfile1.q failed due to script file not found, should verify whether add script file to SparkContext. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7775) enable sample8.q.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114353#comment-14114353 ] Szehon Ho commented on HIVE-7775: - This test is still failing on recent spark-builds, can somebody please take a look? Thanks. enable sample8.q.[Spark Branch] --- Key: HIVE-7775 URL: https://issues.apache.org/jira/browse/HIVE-7775 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Fix For: spark-branch Attachments: HIVE-7775.1-spark.patch, HIVE-7775.2-spark.patch sample8.q contain join query, should enable this qtest after hive on spark support join operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
