[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512191#comment-15512191 ] Ferdinand Xu commented on HIVE-14029: - Thanks [~stakiar] for your review. Description is updated. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14029: Description: There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0 to benefit those performance improvements. To update Spark version to 2.0.0, the following changes are required: * Spark API updates: ** SparkShuffler#call return Iterator instead of Iterable ** SparkListener -> JavaSparkListener ** InputMetrics constructor doesn’t accept readMethod ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long type instead of integer * Dependency upgrade: ** Jackson: 2.4.2 -> 2.6.5 ** Netty version: 4.0.23.Final -> 4.0.29.Final ** Scala binary version: 2.10 -> 2.11 ** Scala version: 2.10.4 -> 2.11.8 was: There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0 to benefit those performance improvements. To update Spark version to 2.0.0, the following changes are required: *Spark API updates:* SparkShuffler#call return Iterator instead of Iterable SparkListener -> JavaSparkListener InputMetrics constructor doesn’t accept readMethod Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long type instead of integer *Dependency upgrade: Jackson: 2.4.2 -> 2.6.5 Netty version: 4.0.23.Final -> 4.0.29.Final Scala binary version: 2.10 -> 2.11 Scala version: 2.10.4 -> 2.11.8 > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14029: Description: There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0 to benefit those performance improvements. To update Spark version to 2.0.0, the following changes are required: *Spark API updates:* SparkShuffler#call return Iterator instead of Iterable SparkListener -> JavaSparkListener InputMetrics constructor doesn’t accept readMethod Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long type instead of integer *Dependency upgrade: Jackson: 2.4.2 -> 2.6.5 Netty version: 4.0.23.Final -> 4.0.29.Final Scala binary version: 2.10 -> 2.11 Scala version: 2.10.4 -> 2.11.8 was:There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0 to benefit those performance improvements. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > *Spark API updates:* > SparkShuffler#call return Iterator instead of Iterable > SparkListener -> JavaSparkListener > InputMetrics constructor doesn’t accept readMethod > Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > *Dependency upgrade: > Jackson: 2.4.2 -> 2.6.5 > Netty version: 4.0.23.Final -> 4.0.29.Final > Scala binary version: 2.10 -> 2.11 > Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512148#comment-15512148 ] Xuefu Zhang commented on HIVE-14412: +1 > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, > HIVE-14412.6.patch, HIVE-14412.7.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512057#comment-15512057 ] Chaoyu Tang commented on HIVE-14774: [~mohitsabharwal] SQLOperation is running a query in a background thread, its driver does lock acquisition, query execution etc. When Ctrl-C shuts down Beeline, it triggers TServerEventHandler deleteContext which in term closes the Session therefore cleaning up the operation (in the session thread). This cleanup close/destroy Driver (nullify its fields like ctx, plan etc), so the Driver running in background might fail in a certain step by RuntimeException (e.g. NPE) which have not been handled completely or properly in Hive. In a race condition when the Drive is about to acquire the locks in background thread and the SQLOperation cleanup has destroyed/closed driver but not yet kill the ground thread, the query process running in the background could still be able to acquire the locks but these locks might never have a chance to be cleared later. The simplest and right way is first to kill the running query in the background, followed by driver close/destroy which ensures that the locks, if any, acquired in the Driver would be cleared. Does that make sense? > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.7.patch Thank you very much [~xuefuz] for the review. Update patch to address your comments. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, > HIVE-14412.6.patch, HIVE-14412.7.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[
https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512009#comment-15512009
]
Hive QA commented on HIVE-14814:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829731/HIVE-14814.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10554 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1265/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1265/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1265/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829731 - PreCommit-HIVE-Build
> metastoreClient is used directly in Hive cause NPE
> --
>
> Key: HIVE-14814
> URL: https://issues.apache.org/jira/browse/HIVE-14814
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.1.0, 2.2.0
>Reporter: Dileep Kumar Chiguruvada
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14814.1.patch
>
>
> Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java
> which may be null causing NPE. Instead it should use getMSC() which will
> initialize metastoreClient variable when null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511974#comment-15511974 ] Xuefu Zhang commented on HIVE-14412: [~lirui], I have a couple of minor comment on RB. License header is probably a must. Otherwise, patch looks good. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, HIVE-14412.6.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511956#comment-15511956 ] Mohit Sabharwal commented on HIVE-14774: [~ctang.ma], since locks are cleared in driver.destroy(), I was wondering how killing the query process is related to clearing locks. > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511898#comment-15511898
]
Hive QA commented on HIVE-14731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829718/HIVE-14731.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10525 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver-tez_schema_evolution.q-tez_join.q-file_with_header_footer.q-and-27-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_partitioned]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cross_join]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1264/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1264/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1264/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829718 - PreCommit-HIVE-Build
> Use Tez cartesian product edge in Hive (unpartitioned case only)
>
>
> Key: HIVE-14731
> URL: https://issues.apache.org/jira/browse/HIVE-14731
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch,
> HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch,
> HIVE-14731.6.patch, HIVE-14731.7.patch
>
>
> Given cartesian product edge is available in Tez now (see TEZ-3230), let's
> integrate it into Hive on Tez. This allows us to have more than one reducer
> in cross product queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew
[
https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511843#comment-15511843
]
roncenzhao commented on HIVE-14797:
---
I think they are not related to my patch. The failure testcases have run
successfully in my own machine.
> reducer number estimating may lead to data skew
> ---
>
> Key: HIVE-14797
> URL: https://issues.apache.org/jira/browse/HIVE-14797
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: roncenzhao
>Assignee: roncenzhao
> Attachments: HIVE-14797.2.patch, HIVE-14797.3.patch, HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is
> implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
> int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i],
> bucketFieldInspectors[i]);
> hashCode = 31 * hashCode + fieldHash;
> }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b
> string. The values of column 'a' in both two tables are not skew, but values
> of column 'b' in both two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and
> tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data
> skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`.
> When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the
> result, the job will be skew.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14797) reducer number estimating may lead to data skew
[
https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
roncenzhao updated HIVE-14797:
--
Attachment: HIVE-14797.3.patch
Remove some code duplication
> reducer number estimating may lead to data skew
> ---
>
> Key: HIVE-14797
> URL: https://issues.apache.org/jira/browse/HIVE-14797
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: roncenzhao
>Assignee: roncenzhao
> Attachments: HIVE-14797.2.patch, HIVE-14797.3.patch, HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is
> implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
> int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i],
> bucketFieldInspectors[i]);
> hashCode = 31 * hashCode + fieldHash;
> }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b
> string. The values of column 'a' in both two tables are not skew, but values
> of column 'b' in both two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and
> tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data
> skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`.
> When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the
> result, the job will be skew.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[ https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511780#comment-15511780 ] Prasanth Jayachandran commented on HIVE-14814: -- Yeah. Updated. Will backport them. > metastoreClient is used directly in Hive cause NPE > -- > > Key: HIVE-14814 > URL: https://issues.apache.org/jira/browse/HIVE-14814 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Dileep Kumar Chiguruvada >Assignee: Prasanth Jayachandran > Attachments: HIVE-14814.1.patch > > > Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java > which may be null causing NPE. Instead it should use getMSC() which will > initialize metastoreClient variable when null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[ https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14814: - Affects Version/s: 1.3.0 2.1.0 > metastoreClient is used directly in Hive cause NPE > -- > > Key: HIVE-14814 > URL: https://issues.apache.org/jira/browse/HIVE-14814 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Dileep Kumar Chiguruvada >Assignee: Prasanth Jayachandran > Attachments: HIVE-14814.1.patch > > > Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java > which may be null causing NPE. Instead it should use getMSC() which will > initialize metastoreClient variable when null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[ https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14814: - Target Version/s: 2.1.0, 1.3.0, 2.2.0 (was: 2.2.0) > metastoreClient is used directly in Hive cause NPE > -- > > Key: HIVE-14814 > URL: https://issues.apache.org/jira/browse/HIVE-14814 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Dileep Kumar Chiguruvada >Assignee: Prasanth Jayachandran > Attachments: HIVE-14814.1.patch > > > Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java > which may be null causing NPE. Instead it should use getMSC() which will > initialize metastoreClient variable when null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[ https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511757#comment-15511757 ] Eugene Koifman commented on HIVE-14814: --- +1 patch 1 but since HIVE-13622 this would have to go into 2.1.1 and 1.3, right? > metastoreClient is used directly in Hive cause NPE > -- > > Key: HIVE-14814 > URL: https://issues.apache.org/jira/browse/HIVE-14814 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Dileep Kumar Chiguruvada >Assignee: Prasanth Jayachandran > Attachments: HIVE-14814.1.patch > > > Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java > which may be null causing NPE. Instead it should use getMSC() which will > initialize metastoreClient variable when null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511756#comment-15511756
]
Hive QA commented on HIVE-14731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829718/HIVE-14731.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10555 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cross_join]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mergejoin]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1263/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1263/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1263/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829718 - PreCommit-HIVE-Build
> Use Tez cartesian product edge in Hive (unpartitioned case only)
>
>
> Key: HIVE-14731
> URL: https://issues.apache.org/jira/browse/HIVE-14731
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch,
> HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch,
> HIVE-14731.6.patch, HIVE-14731.7.patch
>
>
> Given cartesian product edge is available in Tez now (see TEZ-3230), let's
> integrate it into Hive on Tez. This allows us to have more than one reducer
> in cross product queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
[
https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511742#comment-15511742
]
Rui Li edited comment on HIVE-14719 at 9/22/16 1:17 AM:
Guess [~ashutoshc] is busy with other tasks. [~jcamachorodriguez],
[~hsubramaniyan] could you take a look at this? Thanks.
was (Author: lirui):
Guess [~ashutoshc] is busy with other tasks. [~ jcamachorodriguez], [~
hsubramaniyan] could you take a look at this? Thanks.
> ASTNode rootNode is not maintained properly when changing child/parent
> relation
> ---
>
> Key: HIVE-14719
> URL: https://issues.apache.org/jira/browse/HIVE-14719
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14719.1.patch, HIVE-14719.2.patch
>
>
> When I run some query like:
> {code}
> set hive.cbo.enable=false;
> select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
> {code}
> It gets error like:
> {noformat}
> FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
> 'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
> (tok_table_or_col sq_1) sq_corr_1))
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
[
https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511742#comment-15511742
]
Rui Li commented on HIVE-14719:
---
Guess [~ashutoshc] is busy with other tasks. [~ jcamachorodriguez], [~
hsubramaniyan] could you take a look at this? Thanks.
> ASTNode rootNode is not maintained properly when changing child/parent
> relation
> ---
>
> Key: HIVE-14719
> URL: https://issues.apache.org/jira/browse/HIVE-14719
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14719.1.patch, HIVE-14719.2.patch
>
>
> When I run some query like:
> {code}
> set hive.cbo.enable=false;
> select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
> {code}
> It gets error like:
> {noformat}
> FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
> 'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
> (tok_table_or_col sq_1) sq_corr_1))
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511722#comment-15511722 ] Rui Li commented on HIVE-14412: --- Latest failures not related and I've uploaded v6 to RB. [~xuefuz] please take another look. Thanks. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, HIVE-14412.6.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew
[
https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511708#comment-15511708
]
Rui Li commented on HIVE-14797:
---
I see some failures "did not produce a TEST-*.xml file". Are they related?
> reducer number estimating may lead to data skew
> ---
>
> Key: HIVE-14797
> URL: https://issues.apache.org/jira/browse/HIVE-14797
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: roncenzhao
>Assignee: roncenzhao
> Attachments: HIVE-14797.2.patch, HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is
> implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
> int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i],
> bucketFieldInspectors[i]);
> hashCode = 31 * hashCode + fieldHash;
> }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b
> string. The values of column 'a' in both two tables are not skew, but values
> of column 'b' in both two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and
> tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data
> skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`.
> When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the
> result, the job will be skew.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[ https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14814: - Reporter: Dileep Kumar Chiguruvada (was: Prasanth Jayachandran) > metastoreClient is used directly in Hive cause NPE > -- > > Key: HIVE-14814 > URL: https://issues.apache.org/jira/browse/HIVE-14814 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Dileep Kumar Chiguruvada >Assignee: Prasanth Jayachandran > Attachments: HIVE-14814.1.patch > > > Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java > which may be null causing NPE. Instead it should use getMSC() which will > initialize metastoreClient variable when null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[ https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14814: - Status: Patch Available (was: Open) > metastoreClient is used directly in Hive cause NPE > -- > > Key: HIVE-14814 > URL: https://issues.apache.org/jira/browse/HIVE-14814 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14814.1.patch > > > Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java > which may be null causing NPE. Instead it should use getMSC() which will > initialize metastoreClient variable when null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14814) metastoreClient is used directly in Hive cause NPE
[ https://issues.apache.org/jira/browse/HIVE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14814: - Attachment: HIVE-14814.1.patch [~ekoifman] Can you please review this change as it is caused by HIVE-13622 changes? > metastoreClient is used directly in Hive cause NPE > -- > > Key: HIVE-14814 > URL: https://issues.apache.org/jira/browse/HIVE-14814 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14814.1.patch > > > Changes introduced by HIVE-13622 uses metastoreClient directly in Hive.java > which may be null causing NPE. Instead it should use getMSC() which will > initialize metastoreClient variable when null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511682#comment-15511682 ] Rui Li commented on HIVE-14029: --- [~Ferd], the classpath is just for HS2/CLI, so I don't think we need all the spark jars. Please find a minimum set of required jars. You can start with spark-core. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14802) bogus HiveMetaStore.getDelegationToken calls in HS2 session opening
[
https://issues.apache.org/jira/browse/HIVE-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511644#comment-15511644
]
Sergey Shelukhin commented on HIVE-14802:
-
[~sushanth] [~thejas] do you know what this is?
> bogus HiveMetaStore.getDelegationToken calls in HS2 session opening
> ---
>
> Key: HIVE-14802
> URL: https://issues.apache.org/jira/browse/HIVE-14802
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Not sure what's going on. Tons of these in some log, looks like an internal
> error {noformat}
> 2016-09-07 07:57:27,226 ERROR [HiveServer2-HttpHandler-Pool: Thread-22520]:
> metadata.Hive (Hive.java:getDelegationToken(2890)) -
> java.lang.UnsupportedOperationException: getDelegationToken() can be called
> only in thrift (non local) mode
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:1664)
> at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
> at com.sun.proxy.$Proxy8.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2888)
> at
> org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:468)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getDelegationToken(ThriftCLIService.java:340)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:321)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:235)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:101)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511614#comment-15511614 ] Ferdinand Xu commented on HIVE-14029: - Hi [~spena], do we need to support in Hive 2.1? I do some smoke tests in current upstream and Spark 2.0 and it passed if you set SPARK_HOME correctly and copy all lib jars of Spark into hive/lib folder. This needed to be updated in Hive On Spark WIKI. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511610#comment-15511610 ] Ferdinand Xu commented on HIVE-14029: - Hi [~xuefuz], Spark assembly was removed since Spark 2.0.0. They don't provide an assembly jar considering some dependency conflicts. I find some comments in the root pom file for Spark. To support 2.0.0, we have to copy all Spark related jars under the hive/lib AFAIK. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[
https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511583#comment-15511583
]
Hive QA commented on HIVE-14774:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829705/HIVE-14774.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10554 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1262/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1262/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1262/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829705 - PreCommit-HIVE-Build
> Canceling query using Ctrl-C in beeline might lead to stale locks
> -
>
> Key: HIVE-14774
> URL: https://issues.apache.org/jira/browse/HIVE-14774
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-14774.patch
>
>
> Terminating a running query using Ctrl-C in Beeline might lead to stale locks
> since the process running the query might still be able to acquire the locks
> but fail to release them after the query terminate abnormally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511577#comment-15511577 ] Xuefu Zhang commented on HIVE-14029: I made a build of spark 2.0 and indeed spark-assembly.jar is missing. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511533#comment-15511533 ] Xuefu Zhang commented on HIVE-14029: Hi guys, thanks for working/reviewing this. The patch looks good. I understand that there is a pending discussion about removing spark tarball from the test. However, in this long thread there seems a confusion of this with the spark's assembly jar which is part of spark build as of 1.6. [~Ferd], do we have a clear picture of that for 2.0? If there is any change, we do want to update the doc. For instance, I used to get the assembly.jar from spark build and copy it to hive's /lib directory and I'm ready to run Hive on Spark. Sorry I'm a little behind Spark 2.0. I will try to figure it out on my end as well. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511503#comment-15511503 ] Jimmy Xiang commented on HIVE-14774: +1 > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.7.patch rebase patch to fix build failure > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511470#comment-15511470
]
Hive QA commented on HIVE-14731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829708/HIVE-14731.6.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1261/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1261/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1261/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-09-21 23:04:00.438
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1261/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-09-21 23:04:00.442
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
91082e5..ed82cfa master -> origin/master
+ git reset --hard HEAD
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at 91082e5 HIVE-14461. Move hbase_bulk to run via TestCliDriver.
(Siddharth Seth, reviewed by Prasanth Jayachandran)
+ git clean -f -d
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
Removing
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java.orig
+ git checkout master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at ed82cfa HIVE-14783 : bucketing column should be part of sorting
for delete/update operation when spdo is on (Ashutosh Chauhan via Prasanth J)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-09-21 23:04:02.217
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
error: patch failed: itests/src/test/resources/testconfiguration.properties:444
error: itests/src/test/resources/testconfiguration.properties: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829708 - PreCommit-HIVE-Build
> Use Tez cartesian product edge in Hive (unpartitioned case only)
>
>
> Key: HIVE-14731
> URL: https://issues.apache.org/jira/browse/HIVE-14731
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch,
> HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch
>
>
> Given cartesian product edge is available in Tez now (see TEZ-3230), let's
> integrate it into Hive on Tez. This allows us to have more than one reducer
> in cross product queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14505) Analyze org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching failure
[
https://issues.apache.org/jira/browse/HIVE-14505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511466#comment-15511466
]
Hive QA commented on HIVE-14505:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827505/HIVE-14505.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10554 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1260/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1260/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1260/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827505 - PreCommit-HIVE-Build
> Analyze
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
> failure
>
>
> Key: HIVE-14505
> URL: https://issues.apache.org/jira/browse/HIVE-14505
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-14505.1.patch, HIVE-14505.1.patch
>
>
> Flaky test failure. Fails ~50% of the time locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Reporter: Kavan Suresh (was: Ashutosh Chauhan) > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, > HIVE-14783.4.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.6.patch Fixed testconfiguration.properties > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Open (was: Patch Available) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14774: --- Status: Patch Available (was: Open) [~mohitsabharwal], [~jxiang] could you review the patch? Thanks. > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks Prasanth for review. Current test case takes 2 minute to execute. I will trim it down and add it in a follow-up patch. > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, > HIVE-14783.4.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14774: --- Attachment: HIVE-14774.patch Kill the process running query in the background before close/destroy the Driver instance. Otherwise, under certain race condition, the running process might still access some driver fields (e.g. plan, ctx) which have been nullified in the cleanup process and cause some uncaught runtime errors. Some resources such as locks might also not be cleared properly. > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.5.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511323#comment-15511323
]
Hive QA commented on HIVE-14731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829674/HIVE-14731.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1259/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1259/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1259/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-09-21 22:01:36.013
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1259/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-09-21 22:01:36.015
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
66af764..91082e5 master -> origin/master
+ git reset --hard HEAD
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at 66af764 HIVE-14782: Improve runtime of NegativeMinimrCliDriver
(Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
+ git checkout master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at 91082e5 HIVE-14461. Move hbase_bulk to run via TestCliDriver.
(Siddharth Seth, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-09-21 22:01:37.673
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
error: patch failed: itests/src/test/resources/testconfiguration.properties:444
error: itests/src/test/resources/testconfiguration.properties: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829674 - PreCommit-HIVE-Build
> Use Tez cartesian product edge in Hive (unpartitioned case only)
>
>
> Key: HIVE-14731
> URL: https://issues.apache.org/jira/browse/HIVE-14731
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch,
> HIVE-14731.3.patch, HIVE-14731.4.patch
>
>
> Given cartesian product edge is available in Tez now (see TEZ-3230), let's
> integrate it into Hive on Tez. This allows us to have more than one reducer
> in cross product queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14505) Analyze org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching failure
[ https://issues.apache.org/jira/browse/HIVE-14505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511318#comment-15511318 ] Siddharth Seth commented on HIVE-14505: --- [~thejas] - could you please review the patch? > Analyze > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching > failure > > > Key: HIVE-14505 > URL: https://issues.apache.org/jira/browse/HIVE-14505 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-14505.1.patch, HIVE-14505.1.patch > > > Flaky test failure. Fails ~50% of the time locally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[
https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511315#comment-15511315
]
Hive QA commented on HIVE-14783:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829650/HIVE-14783.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10554 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1258/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1258/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1258/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829650 - PreCommit-HIVE-Build
> bucketing column should be part of sorting for delete/update operation when
> spdo is on
> --
>
> Key: HIVE-14783
> URL: https://issues.apache.org/jira/browse/HIVE-14783
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Transactions
>Affects Versions: 2.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.2.0
>
> Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch,
> HIVE-14783.4.patch, HIVE-14783.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-14691) Investigate removal of TestHBaseMinimrCliDriver
[ https://issues.apache.org/jira/browse/HIVE-14691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-14691. --- Resolution: Duplicate > Investigate removal of TestHBaseMinimrCliDriver > --- > > Key: HIVE-14691 > URL: https://issues.apache.org/jira/browse/HIVE-14691 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth > > Follow up from discussion on HIVE-14576. Look at whether this test can be > completely removed given the MiniHBaseCliDriver tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14461) Move hbase_bulk to run via TestCliDriver.
[ https://issues.apache.org/jira/browse/HIVE-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14461: -- Parent Issue: HIVE-14547 (was: HIVE-14443) > Move hbase_bulk to run via TestCliDriver. > - > > Key: HIVE-14461 > URL: https://issues.apache.org/jira/browse/HIVE-14461 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14461.01.patch > > > during HIVE-1 i've encountered an odd thing: > HBaseMinimrCliDriver only executes single test...and that test is set using > the qfile selector...which looks a out-of-place. > The only test it executes doesn't follow regular qtest file naming...and has > an extension 'm' > At least the file should be renamedbut I think change wasn't > intentional -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14461) Move hbase_bulk to run via TestCliDriver.
[ https://issues.apache.org/jira/browse/HIVE-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14461: -- Summary: Move hbase_bulk to run via TestCliDriver. (was: Investigate HBaseMinimrCliDriver tests) > Move hbase_bulk to run via TestCliDriver. > - > > Key: HIVE-14461 > URL: https://issues.apache.org/jira/browse/HIVE-14461 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14461.01.patch > > > during HIVE-1 i've encountered an odd thing: > HBaseMinimrCliDriver only executes single test...and that test is set using > the qfile selector...which looks a out-of-place. > The only test it executes doesn't follow regular qtest file naming...and has > an extension 'm' > At least the file should be renamedbut I think change wasn't > intentional -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14461) Investigate HBaseMinimrCliDriver tests
[ https://issues.apache.org/jira/browse/HIVE-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14461: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks for the review. Committed. > Investigate HBaseMinimrCliDriver tests > -- > > Key: HIVE-14461 > URL: https://issues.apache.org/jira/browse/HIVE-14461 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14461.01.patch > > > during HIVE-1 i've encountered an odd thing: > HBaseMinimrCliDriver only executes single test...and that test is set using > the qfile selector...which looks a out-of-place. > The only test it executes doesn't follow regular qtest file naming...and has > an extension 'm' > At least the file should be renamedbut I think change wasn't > intentional -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14808) Support TRUNCATE in tez
[ https://issues.apache.org/jira/browse/HIVE-14808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14808: - Assignee: (was: Prasanth Jayachandran) > Support TRUNCATE in tez > --- > > Key: HIVE-14808 > URL: https://issues.apache.org/jira/browse/HIVE-14808 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran > > Truncate query is MR only at this point. Add support for TRUNCATE in tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511087#comment-15511087
]
Hive QA commented on HIVE-14805:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829640/HIVE-14805.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10555 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_union_view]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_union_view]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_view]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testMetadataQueriesWithSerializeThriftInTasks
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation2
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1257/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1257/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1257/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829640 - PreCommit-HIVE-Build
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14805.1.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14098: --- Resolution: Fixed Fix Version/s: 2.1.1 Status: Resolved (was: Patch Available) Thanks [~pvary]. I committed to branch-2.1 > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Affects Versions: 2.1.0 >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14098-branch-2.1.patch, > HIVE-14098.2-branch-2.1.patch, HIVE-14098.2.patch, > HIVE-14098.3-branch-2.1.patch, HIVE-14098.4-branch-2.1.patch, HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511038#comment-15511038
]
Sergio Peña commented on HIVE-14029:
[~Ferd] The patch looks good. +1
I just found a variable that is not used anymore.
{noformat}
MetricsCollection.java
- Should we remove 'DataReadMethod readMethod = null'? is not used anymore.
{noformat}
[~xuefuz] Do you think this patch is ready to go to start supporting spark 2.0?
[~Ferd] Have we run any other tests in an environment with spark 2.0 and hive
2.1? I think we should do that if you haven't yet before commit the patch. Just
to confirm we don't have issues with the classpath.
> Update Spark version to 2.0.0
> -
>
> Key: HIVE-14029
> URL: https://issues.apache.org/jira/browse/HIVE-14029
> Project: Hive
> Issue Type: Bug
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch,
> HIVE-14029.3.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up
> Spark to 2.0.0 to benefit those performance improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.4.patch All failed tests beyond testCliDriver[dynamic_partition_pruning] are not age 1. Also there is no dynamic_partition_pruning test for testCliDriver, no sure why this test was ran. Reupload patch to trigger test run for debugging. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14806) Support UDTF in CBO (AST return path)
[ https://issues.apache.org/jira/browse/HIVE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510981#comment-15510981 ] Pengcheng Xiong commented on HIVE-14806: [~ashutoshc]. The test cases failures are OK and they actually proved that the new UDTF code path works. > Support UDTF in CBO (AST return path) > - > > Key: HIVE-14806 > URL: https://issues.apache.org/jira/browse/HIVE-14806 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14806.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Implement some admission control mechanism for graceful degradation when resources are exhausted
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510911#comment-15510911
]
Lefty Leverenz commented on HIVE-9423:
--
For hs2-unexpected-end-of-file, I suggest two changes:
* "too high concurrent number of connections" -> "too many concurrent
connections"
* "is applicable." -> "if applicable."
For hs2-could-not-open-connection, the comma in the second sentence isn't
needed (but it does no harm so keep it if you like).
For hs2-unknown-connection-problem & hs2-unexpected-error, Thrift should have
an initial capital letter.
> HiveServer2: Implement some admission control mechanism for graceful
> degradation when resources are exhausted
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14461) Investigate HBaseMinimrCliDriver tests
[ https://issues.apache.org/jira/browse/HIVE-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510901#comment-15510901 ] Prasanth Jayachandran commented on HIVE-14461: -- +1 > Investigate HBaseMinimrCliDriver tests > -- > > Key: HIVE-14461 > URL: https://issues.apache.org/jira/browse/HIVE-14461 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Siddharth Seth > Attachments: HIVE-14461.01.patch > > > during HIVE-1 i've encountered an odd thing: > HBaseMinimrCliDriver only executes single test...and that test is set using > the qfile selector...which looks a out-of-place. > The only test it executes doesn't follow regular qtest file naming...and has > an extension 'm' > At least the file should be renamedbut I think change wasn't > intentional -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14806) Support UDTF in CBO (AST return path)
[
https://issues.apache.org/jira/browse/HIVE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510879#comment-15510879
]
Hive QA commented on HIVE-14806:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829633/HIVE-14806.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10553 tests
executed
*Failed tests:*
{noformat}
296_TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_noalias]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_udtf]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_inline]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_explode]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_json_tuple]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_parse_url_tuple]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1256/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1256/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1256/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829633 - PreCommit-HIVE-Build
> Support UDTF in CBO (AST return path)
> -
>
> Key: HIVE-14806
> URL: https://issues.apache.org/jira/browse/HIVE-14806
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14806.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14793) Allow ptest branch to be specified, PROFILE override
[ https://issues.apache.org/jira/browse/HIVE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510871#comment-15510871 ] Siddharth Seth commented on HIVE-14793: --- Probably better with all of 14744 in place. This adds 3 optional parameters PTEST_GIT_REPO PTEST_GIT_BRANCH BUILD_PROFILE > Allow ptest branch to be specified, PROFILE override > > > Key: HIVE-14793 > URL: https://issues.apache.org/jira/browse/HIVE-14793 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14793.01.patch, HIVE-14793.02.patch, > HIVE-14793.03.patch > > > Post HIVE-14734 - the profile is automatically determined. Add an option to > override this via Jenkins. Also add an option to specify the branch from > which ptest is built (This is hardcoded to github.com/apache/hive) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14782) Improve runtime of NegativeMinimrCliDriver
[ https://issues.apache.org/jira/browse/HIVE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510845#comment-15510845 ] Siddharth Seth commented on HIVE-14782: --- +1 > Improve runtime of NegativeMinimrCliDriver > -- > > Key: HIVE-14782 > URL: https://issues.apache.org/jira/browse/HIVE-14782 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14782.1.patch, HIVE-14782.2.patch, > HIVE-14782.3.patch > > > NegativeMinimrCliDriver is one of the slowest test batch. The actual test > takes only 3 minutes where as initialization of test takes around 15 minutes. > Also remove hadoop20.q tests from NegativeMinimrCliDriver batch as it is no > longer supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510832#comment-15510832 ] Prasanth Jayachandran commented on HIVE-14783: -- Are the test failures related? The new changes lgtm. Can you add some new test cases that covers these failures? to make sure delete is working with and without SPDO. > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, > HIVE-14783.4.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Status: Patch Available (was: Open) > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, > HIVE-14783.4.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Status: Open (was: Patch Available) > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, > HIVE-14783.4.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Attachment: HIVE-14783.4.patch > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, > HIVE-14783.4.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10685) Alter table concatenate oparetor will cause duplicate data
[
https://issues.apache.org/jira/browse/HIVE-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510822#comment-15510822
]
Prasanth Jayachandran commented on HIVE-10685:
--
It got committed
https://github.com/apache/hive/commit/aef08f44e29e9a54e73b8029892033fe16c52cc5
> Alter table concatenate oparetor will cause duplicate data
> --
>
> Key: HIVE-10685
> URL: https://issues.apache.org/jira/browse/HIVE-10685
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 1.2.1
>Reporter: guoliming
>Assignee: guoliming
>Priority: Critical
> Fix For: 1.2.1
>
> Attachments: HIVE-10685.patch, HIVE-10685.patch
>
>
> "Orders" table has 15 rows and stored as ORC.
> {noformat}
> hive> select count(*) from orders;
> OK
> 15
> Time taken: 37.692 seconds, Fetched: 1 row(s)
> {noformat}
> The table contain 14 files,the size of each file is about 2.1 ~ 3.2 GB.
> After executing command : ALTER TABLE orders CONCATENATE;
> The table is already 1530115000 rows.
> My hive version is 1.1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510742#comment-15510742
]
Aihua Xu commented on HIVE-14805:
-
[~thejas] Can you help me review it? It's similar to HIVE-10875.
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14805.1.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13703) "msck repair" on table with non-partition subdirectories reporting partitions not in metastore
[
https://issues.apache.org/jira/browse/HIVE-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin resolved HIVE-13703.
-
Resolution: Duplicate
> "msck repair" on table with non-partition subdirectories reporting partitions
> not in metastore
> --
>
> Key: HIVE-13703
> URL: https://issues.apache.org/jira/browse/HIVE-13703
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.14.0, 1.0.0, 1.2.1
>Reporter: Ana Gillan
>Assignee: Alina Abramova
> Attachments: HIVE-13703.patch
>
>
> PROBLEM: Subdirectories created with UNION ALL are listed in {{show
> partitions}} output, but show up as {{Partitions not in metastore}} in {{msck
> repair}} output.
> STEPS TO REPRODUCE: Table created from {{CTAS ... UNION ALL}} DDL
> {code}
> hive> msck repair table meter_001;
> OK
> Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1
> meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1
> meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1
> meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1
> meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1
> meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1
> meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1
> meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1
> meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1
> meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1
> meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1
> meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1
> meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1
> meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1
> meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1
> meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1
> meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1
> meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1
> meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1
> meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1
> meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1
> meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1
> meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1
> meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1
> meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1
> meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1
> meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1
> meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1
> meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1
> meter_001:tech_datestamp=2016-04-06/2
> Time taken: 15.996 seconds, Fetched: 1 row(s)
> {code}
> {code}
> hive> show partitions meter_001;
> OK
> tech_datestamp=2016-03-09
> tech_datestamp=2016-03-10
> tech_datestamp=2016-03-11
> tech_datestamp=2016-03-12
> tech_datestamp=2016-03-13
> tech_datestamp=2016-03-14
> tech_datestamp=2016-03-15
> tech_datestamp=2016-03-16
> tech_datestamp=2016-03-17
> tech_datestamp=2016-03-18
> tech_datestamp=2016-03-19
> tech_datestamp=2016-03-20
> tech_datestamp=2016-03-21
> tech_datestamp=2016-03-22
> tech_datestamp=2016-03-23
> tech_datestamp=2016-03-24
> tech_datestamp=2016-03-25
> tech_datestamp=2016-03-26
> tech_datestamp=2016-03-27
> tech_datestamp=2016-03-28
> tech_datestamp=2016-03-29
> tech_datestamp=2016-03-30
> tech_datestamp=2016-03-31
> tech_datestamp=2016-04-01
> tech_datestamp=2016-04-02
> tech_datestamp=2016-04-03
> tech_datestamp=2016-04-04
> tech_datestamp=2016-04-05
> tech_datestamp=2016-04-06
> Time taken: 0.417 seconds, Fetched: 29 row(s)
> {code}
> Ideally msck repair should ignore subdirectory if that additional partition
> column doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510717#comment-15510717
]
Aihua Xu commented on HIVE-14805:
-
patch-1: we are not passing insideView from the current queryblock to the
subquery.
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14805.1.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Status: Patch Available (was: Open)
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14805.1.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Attachment: HIVE-14805.1.patch
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14805.1.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Attachment: (was: HIVE-14805.1.patch)
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14805.1.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[
https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510711#comment-15510711
]
Hive QA commented on HIVE-14783:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829617/HIVE-14783.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10556 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_nullvalues]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_nullsafe]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[special_character_in_tabnames_1]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[join_nullsafe]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_nullsafe]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1255/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1255/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1255/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829617 - PreCommit-HIVE-Build
> bucketing column should be part of sorting for delete/update operation when
> spdo is on
> --
>
> Key: HIVE-14783
> URL: https://issues.apache.org/jira/browse/HIVE-14783
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Transactions
>Affects Versions: 2.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.2.0
>
> Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, HIVE-14783.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Attachment: HIVE-14805.1.patch
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14805.1.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Summary: Subquery inside a view will have the object in the subquery as the
direct input (was: Subquery inside a view will have the object in the
subquery as the input )
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will have the object in the subquery as the input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Summary: Subquery inside a view will have the object in the subquery as the
input (was: Subquery inside a view will treat the object in the subquery as
the input )
> Subquery inside a view will have the object in the subquery as the input
> -
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14806) Support UDTF in CBO (AST return path)
[ https://issues.apache.org/jira/browse/HIVE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14806: --- Status: Patch Available (was: Open) > Support UDTF in CBO (AST return path) > - > > Key: HIVE-14806 > URL: https://issues.apache.org/jira/browse/HIVE-14806 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14806.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will treat the object in the subquery as the input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Summary: Subquery inside a view will treat the object in the subquery as
the input (was: Subquery inside a view doesn't set InsideView property
correctly)
> Subquery inside a view will treat the object in the subquery as the input
> --
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14806) Support UDTF in CBO (AST return path)
[ https://issues.apache.org/jira/browse/HIVE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14806: --- Attachment: HIVE-14806.01.patch > Support UDTF in CBO (AST return path) > - > > Key: HIVE-14806 > URL: https://issues.apache.org/jira/browse/HIVE-14806 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14806.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14806) Support UDTF in CBO (AST return path)
[ https://issues.apache.org/jira/browse/HIVE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510690#comment-15510690 ] Pengcheng Xiong commented on HIVE-14806: [~ashutoshc], could u take a look? I did not add more tests as I assume that the existing tests are sufficient, e.g., explode, stack, etc. > Support UDTF in CBO (AST return path) > - > > Key: HIVE-14806 > URL: https://issues.apache.org/jira/browse/HIVE-14806 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14806.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510656#comment-15510656 ] Sahil Takiar commented on HIVE-14029: - [~Ferd] overall this looks good to me. It would be nice if you could update the description to list out the high-level changes that needed to be made to Hive to add support for Spark 2.0.0. For example, dependency updates, which APIs changed (change from Iterable to Iterator, InputMetrics constructor change). > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view doesn't set InsideView property correctly
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Description:
Here is the repro steps.
{noformat}
create table t1(col string);
create view v1 as select * from t1;
create view dataview as select * from (select * from v1) v2;
select * from dataview;
{noformat}
If hive is configured with authorization hook like Sentry, it will require the
access not only for dataview but also for v1, which should not be required.
The subquery seems to not carry insideview property from the parent query.
was:
Here is the repro steps.
{noformat}
create table t1(col string);
create view v1 as select * from t1;
create view dataview as select v1.col from v1 join (select * from v1) v2 on
v1.col=v2.col;
select * from dataview;
{noformat}
If hive is configured with authorization hook like Sentry, it will require the
access not only for dataview but also for v1, which should not be required.
The subquery seems to not carry insideview property from the parent query.
> Subquery inside a view doesn't set InsideView property correctly
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14240) HoS itests shouldn't depend on a Spark distribution
[
https://issues.apache.org/jira/browse/HIVE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510628#comment-15510628
]
Sahil Takiar commented on HIVE-14240:
-
[~Ferd], [~lirui] yes I forgot that there are two ways qtests get run on spark,
one is in local-cluster mode and the other is in yarn-client mode. I believe
the dependency on a SPARK_HOME directory is present in both modes. So unless we
can figure out a way to change this in Spark, I think we still need the
dependency on the Spark distribution.
> HoS itests shouldn't depend on a Spark distribution
> ---
>
> Key: HIVE-14240
> URL: https://issues.apache.org/jira/browse/HIVE-14240
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> The HoS integration tests download a full Spark Distribution (a tar-ball)
> from CloudFront. It uses this distribution to run Spark locally. It runs a
> few tests with Spark in embedded mode, and some tests against a local Spark
> on YARN cluster. The {{itests/pom.xml}} actually contains scripts to download
> the tar-ball from a pre-defined location.
> This is problematic because the Spark Distribution shades all its
> dependencies, including Hadoop dependencies. This can cause problems when
> upgrading the Hadoop version for Hive (ref: HIVE-13930).
> Removing it will also avoid having to download the tar-ball during every
> build, and simplify the build process for the itests module.
> The Hive itests should instead directly depend on Spark artifacts published
> in Maven Central. It will require some effort to get this working. The
> current Hive Spark Client uses a launch script in the Spark installation to
> run Spark jobs. The script basically does some setup work and invokes
> org.apache.spark.deploy.SparkSubmit. It is possible to invoke this class
> directly, which avoids the need to have a full Spark distribution available
> locally (in fact this option already exists, but isn't tested).
> There may be other issues around classpath conflicts between Hive and Spark.
> For example, Hive and Spark require different versions of Kyro. One solution
> to this would be to take Spark artifacts and shade Kyro inside them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Attachment: HIVE-14783.3.patch > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Status: Patch Available (was: Open) > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.3.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14783) bucketing column should be part of sorting for delete/update operation when spdo is on
[ https://issues.apache.org/jira/browse/HIVE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14783: Status: Open (was: Patch Available) > bucketing column should be part of sorting for delete/update operation when > spdo is on > -- > > Key: HIVE-14783 > URL: https://issues.apache.org/jira/browse/HIVE-14783 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer, Transactions >Affects Versions: 2.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-14783.1.patch, HIVE-14783.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-13589: --- Status: Open (was: Patch Available) > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Vihang Karajgaonkar > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, > HIVE-13589.6.patch, HIVE-13589.7.patch, HIVE-13589.8.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510505#comment-15510505 ] Vihang Karajgaonkar edited comment on HIVE-13589 at 9/21/16 4:56 PM: - @Ke Jia .. I will take it up. Thanks for your consideration. was (Author: vihangk1): Thanks @Ke > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, > HIVE-13589.6.patch, HIVE-13589.7.patch, HIVE-13589.8.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-13589: -- Assignee: Vihang Karajgaonkar (was: Ke Jia) > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Vihang Karajgaonkar > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, > HIVE-13589.6.patch, HIVE-13589.7.patch, HIVE-13589.8.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510505#comment-15510505 ] Vihang Karajgaonkar commented on HIVE-13589: Thanks @Ke > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, > HIVE-13589.6.patch, HIVE-13589.7.patch, HIVE-13589.8.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510503#comment-15510503
]
Hive QA commented on HIVE-14029:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829601/HIVE-14029.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10556 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1254/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1254/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1254/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829601 - PreCommit-HIVE-Build
> Update Spark version to 2.0.0
> -
>
> Key: HIVE-14029
> URL: https://issues.apache.org/jira/browse/HIVE-14029
> Project: Hive
> Issue Type: Bug
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch,
> HIVE-14029.3.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up
> Spark to 2.0.0 to benefit those performance improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view doesn't set InsideView property correctly
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Description:
Here is the repro steps.
{noformat}
create table t1(col string);
create view v1 as select * from t1;
create view dataview as select v1.col from v1 join (select * from v1) v2 on
v1.col=v2.col;
select * from dataview;
{noformat}
If hive is configured with authorization hook like Sentry, it will require the
access not only for dataview but also for v1, which should not be required.
The subquery seems to not carry insideview property from the parent query.
was:
Here is the repro steps.
create table t1(col string);
create view v1 as select * from t1;
create view dataview as select v1.col from v1 join (select * from v1) v2 on
v1.col=v2.col;
select * from dataview;
If hive is configured with authorization hook like Sentry, it will require the
access not only for dataview but also for v1, which should not be required.
The subquery seems to not carry insideview property from the parent query.
> Subquery inside a view doesn't set InsideView property correctly
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select v1.col from v1 join (select * from v1) v2 on
> v1.col=v2.col;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14309) Fix naming of classes in orc module to not conflict with standalone orc
[ https://issues.apache.org/jira/browse/HIVE-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510407#comment-15510407 ] Owen O'Malley commented on HIVE-14309: -- I've had multiple users complain about this now. :( Managing classpaths is *really* error-prone in a lot of different contexts. None of the solutions are great, but this one makes it very clear what happened. We move all of classes that hive publishes out of org.apache.orc to orc.apache.hive.orc. > Fix naming of classes in orc module to not conflict with standalone orc > --- > > Key: HIVE-14309 > URL: https://issues.apache.org/jira/browse/HIVE-14309 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > > The current Hive 2.0 and 2.1 releases have classes in the org.apache.orc > namespace that clash with the ORC project's classes. From Hive 2.2 onward, > the classes will only be on ORC, but we'll reduce the problems of classpath > issues if we rename the classes to org.apache.hive.orc. > I've looked at a set of projects (pig, spark, oozie, flume, & storm) and > can't find any uses of Hive's versions of the org.apache.orc classes, so I > believe this is a safe change that will reduce the integration problems down > stream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510337#comment-15510337 ] Ferdinand Xu commented on HIVE-14029: - Attached as HIVE-14029.3.patch. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew
[
https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510378#comment-15510378
]
Xuefu Zhang commented on HIVE-14797:
The new change seems good. Minor nit: can we change the implementation of
getBucketHashCode() to call the new method with a seed of 31. This is to save
some code duplication.
> reducer number estimating may lead to data skew
> ---
>
> Key: HIVE-14797
> URL: https://issues.apache.org/jira/browse/HIVE-14797
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: roncenzhao
>Assignee: roncenzhao
> Attachments: HIVE-14797.2.patch, HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is
> implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
> int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i],
> bucketFieldInspectors[i]);
> hashCode = 31 * hashCode + fieldHash;
> }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b
> string. The values of column 'a' in both two tables are not skew, but values
> of column 'b' in both two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and
> tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data
> skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`.
> When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the
> result, the job will be skew.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14029: Attachment: HIVE-14029.3.patch > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Implement some admission control mechanism for graceful degradation when resources are exhausted
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510334#comment-15510334
]
Aihua Xu commented on HIVE-9423:
I see. Yeah. Seems good to add the meaningful message for your case if it's
null.
> HiveServer2: Implement some admission control mechanism for graceful
> degradation when resources are exhausted
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510305#comment-15510305
]
Hive QA commented on HIVE-14412:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829579/HIVE-14412.6.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10563 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1253/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1253/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1253/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829579 - PreCommit-HIVE-Build
> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch,
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, HIVE-14412.6.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Implement some admission control mechanism for graceful degradation when resources are exhausted
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510256#comment-15510256
]
Peter Vary commented on HIVE-9423:
--
[~leftylev]: Could you please review the error messages for English correctness?
Thanks,
Peter
> HiveServer2: Implement some admission control mechanism for graceful
> degradation when resources are exhausted
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510257#comment-15510257 ] Sergio Peña commented on HIVE-14029: It is the same as before. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
