[jira] [Updated] (HIVE-17360) Tez session reopen appears to use a wrong conf object

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17360:

Status: Patch Available  (was: Open)

> Tez session reopen appears to use a wrong conf object
> -
>
> Key: HIVE-17360
> URL: https://issues.apache.org/jira/browse/HIVE-17360
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17360.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17360) Tez session reopen appears to use a wrong conf object

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17360:

Attachment: HIVE-17360.patch

[~sseth] this resolves one of the todos you introduced ;) can you take a look 
if this makes sense

> Tez session reopen appears to use a wrong conf object
> -
>
> Key: HIVE-17360
> URL: https://issues.apache.org/jira/browse/HIVE-17360
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17360.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17360) Tez session reopen appears to use a wrong conf object

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-17360:
---


> Tez session reopen appears to use a wrong conf object
> -
>
> Key: HIVE-17360
> URL: https://issues.apache.org/jira/browse/HIVE-17360
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17297) allow AM to use LLAP guaranteed tasks

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133892#comment-16133892
 ] 

Sergey Shelukhin commented on HIVE-17297:
-

[~sseth] do you want to review? These are the changes to AM LLAP scheduler. 
Most of the patch is actually protobuf, Hadoop security fluff, and tests.

> allow AM to use LLAP guaranteed tasks
> -
>
> Key: HIVE-17297
> URL: https://issues.apache.org/jira/browse/HIVE-17297
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17297.01.patch, HIVE-17297.only.patch, 
> HIVE-17297.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17297) allow AM to use LLAP guaranteed tasks

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17297:

Attachment: HIVE-17297.01.patch

Updated the patch

> allow AM to use LLAP guaranteed tasks
> -
>
> Key: HIVE-17297
> URL: https://issues.apache.org/jira/browse/HIVE-17297
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17297.01.patch, HIVE-17297.only.patch, 
> HIVE-17297.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17277) HiveMetastoreClient Log name is wrong

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17277:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Patch 3 committed.  Thanks Zac.

> HiveMetastoreClient Log name is wrong
> -
>
> Key: HIVE-17277
> URL: https://issues.apache.org/jira/browse/HIVE-17277
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-17277.2.patch, HIVE-17277.3.patch, HIVE-17277.patch
>
>
> The name of Log for HiveMetastoreClient is "hive.metastore". It's confused 
> for users to trace hive log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17359) Deal with TypeInfo dependencies in the metastore

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133804#comment-16133804
 ] 

Alan Gates commented on HIVE-17359:
---

Unfortunately there's a three way circular dependency between TypeInfo, SerDe, 
and ObjectInspector.  So the solution to this will need to account for what 
happens to all three of those classes.

The metastore uses TypeInfo in 3 ways and SerDe in one:
# the type names in serdeConstants.java
# the type groupings (e.g. string types, numeric types, ...), also from 
serdeConstants
# the allowed column type transitions in HiveAlterHandler uses 
TypeInfoUtils.implicitConvertible to determine if an "alter table change column 
type" is legal
# HiveMetaStore.get_fields_with_environment_context uses the serde to determine 
the schema for tables where the schema is defined in the storage rather than in 
the metadata (e.g. Avro).

For the purpose of this JIRA I'm only resolving the TypeInfo issues.  We'll 
solve the SerDe issue later, though obviously the choice we make here will 
affect that case.

I see three possible solutions:
# Move the serde package into storage-api.  This would allow the standalone 
metastore (as well as ORC, Parquet, others) to depend on it.  The smaller we 
keep the storage-api the better.  This would bring a lot of code and 
dependencies into it.  Thus I see this as an option of last resort.
# Untangle the TypeInfo, SerDe, ObjectInspector dependency triangle and then 
put TypeInfo into the metastore.  Clean, non-circular dependencies are nice.  
And having type definitions in the metadata makes sense.  But since this would 
change the SerDe and ObjectInspector interfaces it would break every existing 
serde and OI.  I take that to be a non-starter.
#  Duplicate just the needed pieces of TypeInfo in the metastore.  This turns 
out to be a couple hundred lines of code.  Given the stringent backward 
compatibility needs, the odds of type names, type groupings, or alter table 
semantics changing (with the exception of adding new types) seem very low.  The 
downside to this will come in adding new type names, which will required 
changes in hive-serde and the standalone metastore.  On the upside it allows 
the metastore to develop types that Hive might not care about.  I propose to 
take this option.

> Deal with TypeInfo dependencies in the metastore
> 
>
> Key: HIVE-17359
> URL: https://issues.apache.org/jira/browse/HIVE-17359
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> The metastore uses TypeInfo, which resides in the serdes package.  In order 
> to move the metastore to be separately releasable we need to deal with this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17359) Deal with TypeInfo dependencies in the metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates reassigned HIVE-17359:
-


> Deal with TypeInfo dependencies in the metastore
> 
>
> Key: HIVE-17359
> URL: https://issues.apache.org/jira/browse/HIVE-17359
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> The metastore uses TypeInfo, which resides in the serdes package.  In order 
> to move the metastore to be separately releasable we need to deal with this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17307) Change the metastore to not use the metrics code in hive/common

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17307:
--
Status: Patch Available  (was: Open)

> Change the metastore to not use the metrics code in hive/common
> ---
>
> Key: HIVE-17307
> URL: https://issues.apache.org/jira/browse/HIVE-17307
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17307.patch
>
>
> As we move code into the standalone metastore module, it cannot use the 
> metrics in hive-common.  We could copy the current Metrics interface or we 
> could change the metastore code to directly use codahale metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17307) Change the metastore to not use the metrics code in hive/common

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17307:
--
Attachment: HIVE-17307.patch

> Change the metastore to not use the metrics code in hive/common
> ---
>
> Key: HIVE-17307
> URL: https://issues.apache.org/jira/browse/HIVE-17307
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17307.patch
>
>
> As we move code into the standalone metastore module, it cannot use the 
> metrics in hive-common.  We could copy the current Metrics interface or we 
> could change the metastore code to directly use codahale metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17307) Change the metastore to not use the metrics code in hive/common

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133771#comment-16133771
 ] 

ASF GitHub Bot commented on HIVE-17307:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/235

HIVE-17307 Change the metastore to not use the metrics code in hive/common



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17307

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/235.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #235


commit 72fd3e7232a44655dc7471a61729c4a1abf49c26
Author: Alan Gates 
Date:   2017-08-01T16:57:32Z

Implemented metrics in standalone-metastore.  Ported existing metastore 
classes to use them.

commit 201b16defbad4076495553fe127ae7c79e08804b
Author: Alan Gates 
Date:   2017-08-01T16:58:52Z

Adding pom file I missed in the previous checkin.

commit b310893a5f8400085919e313b58723a10fb18c88
Author: Alan Gates 
Date:   2017-08-01T18:36:36Z

Copied JvmPauseMonitor into standalone-metastore

commit 3a8080c6c9cfe77cc0d9561ab5f65d4d4298d5d3
Author: Alan Gates 
Date:   2017-08-01T20:43:14Z

Copied PerfLogger into the standalone metastore and moved 
RetryingHMSHandler to use it.




> Change the metastore to not use the metrics code in hive/common
> ---
>
> Key: HIVE-17307
> URL: https://issues.apache.org/jira/browse/HIVE-17307
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> As we move code into the standalone metastore module, it cannot use the 
> metrics in hive-common.  We could copy the current Metrics interface or we 
> could change the metastore code to directly use codahale metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17343) create a mechanism to get rid of some globals in HS2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133753#comment-16133753
 ] 

Hive QA commented on HIVE-17343:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882677/HIVE-17343.01.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10981 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6461/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6461/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6461/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882677 - PreCommit-HIVE-Build

> create a mechanism to get rid of some globals in HS2
> 
>
> Key: HIVE-17343
> URL: https://issues.apache.org/jira/browse/HIVE-17343
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17343.01.patch, HIVE-17343.patch
>
>
> The intent is to initialize things once in HS2 ctor/init, and then be able to 
> access them from queries, etc. without using globals or threadlocals.
> Things like future workload management work, LLAP coordinator, materialized 
> view registry, etc. could be accessed this way.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17354) Fix "alter view" for incremental replication

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-17354:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Patch pushed to master.

> Fix "alter view" for incremental replication
> 
>
> Key: HIVE-17354
> URL: https://issues.apache.org/jira/browse/HIVE-17354
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Tao Li
>Assignee: Tao Li
> Fix For: 3.0.0
>
> Attachments: HIVE-17354.1.patch
>
>
> There is a bug that "alter view" operation is resulting in a view creation 
> operation instead of a overwriting/replacement operation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17354) Fix "alter view" for incremental replication

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133747#comment-16133747
 ] 

Daniel Dai commented on HIVE-17354:
---

+1

> Fix "alter view" for incremental replication
> 
>
> Key: HIVE-17354
> URL: https://issues.apache.org/jira/browse/HIVE-17354
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17354.1.patch
>
>
> There is a bug that "alter view" operation is resulting in a view creation 
> operation instead of a overwriting/replacement operation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17241) Change metastore classes to not use the shims

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17241:
--
   Resolution: Fixed
 Hadoop Flags: Incompatible change
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Thanks Vaibhav for the review.

Note:  I checked the "incompatible change" flag because of the change in the 
thrift classes.  This should not affect users unless they were implementing 
their own token store.

> Change metastore classes to not use the shims
> -
>
> Key: HIVE-17241
> URL: https://issues.apache.org/jira/browse/HIVE-17241
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 3.0.0
>
> Attachments: HIVE-17241.2.patch, HIVE-17241.patch
>
>
> As part of moving the metastore into a standalone package, it will no longer 
> have access to the shims.  This means we need to either copy them or access 
> the underlying Hadoop operations directly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17241) Change metastore classes to not use the shims

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133744#comment-16133744
 ] 

ASF GitHub Bot commented on HIVE-17241:
---

Github user asfgit closed the pull request at:

https://github.com/apache/hive/pull/228


> Change metastore classes to not use the shims
> -
>
> Key: HIVE-17241
> URL: https://issues.apache.org/jira/browse/HIVE-17241
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17241.2.patch, HIVE-17241.patch
>
>
> As part of moving the metastore into a standalone package, it will no longer 
> have access to the shims.  This means we need to either copy them or access 
> the underlying Hadoop operations directly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-13989) Extended ACLs are not handled according to specification

[Chris Drome (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133742#comment-16133742
 ] 

Chris Drome edited comment on HIVE-13989 at 8/18/17 10:58 PM:
--

For the tests that failed (as opposed to those that timed out), I reran on our 
dev hardware.
I wanted to see if the failure was reproducable and if it also failed at the 
2.2.1 fork point.

|| Test || branch-2.2.1 fork (1ed1f28) || branch-2.2 HEAD + HIVE-13989 ||
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_globallimit] | 
FAILED | FAILED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl] | 
PASSED | PASSED |
| 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_1]
 | FAILED | FAILED |
| 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_compact_binary_search]
 | PASSED | PASSED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[selectindate] | PASSED 
| PASSED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_fast_stats] | 
FAILED | FAILED |
| 
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
| FAILED | FAILED |
| org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
| PASSED | PASSED |
| org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel | 
PASSED | PASSED |
| 
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
 | PASSED | PASSED |
| org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] | PASSED | 
PASSED |
| org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] | PASSED | 
PASSED |

Based on this, HIVE-13989 doesn't appear to be responsible for any of these 
failures.


was (Author: cdrome):
For the tests that failed (as opposed to those that timed out), I reran on our 
dev hardware.
I wanted to see if the failure was reproducable and if it also failed at the 
2.2.1 fork point.

|| Test || branch-2.2.1 fork () || branch-2.2 HEAD + HIVE-13989 ||
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_globallimit] | 
FAILED | FAILED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl] | 
PASSED | PASSED |
| 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_1]
 | FAILED | FAILED |
| 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_compact_binary_search]
 | PASSED | PASSED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[selectindate] | PASSED 
| PASSED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_fast_stats] | 
FAILED | FAILED |
| 
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
| FAILED | FAILED |
| org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
| PASSED | PASSED |
| org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel | 
PASSED | PASSED |
| 
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
 | PASSED | PASSED |
| org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] | PASSED | 
PASSED |
| org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] | PASSED | 
PASSED |

Based on this, HIVE-13989 doesn't appear to be responsible for any of these 
failures.

> Extended ACLs are not handled according to specification
> 
>
> Key: HIVE-13989
> URL: https://issues.apache.org/jira/browse/HIVE-13989
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-13989.1-branch-1.patch, HIVE-13989.1.patch, 
> HIVE-13989.4-branch-2.2.patch, HIVE-13989-branch-1.patch, 
> HIVE-13989-branch-2.2.patch, HIVE-13989-branch-2.2.patch, 
> HIVE-13989-branch-2.2.patch
>
>
> Hive takes two approaches to working with extended ACLs depending on whether 
> data is being produced via a Hive query or HCatalog APIs. A Hive query will 
> run an FsShell command to recursively set the extended ACLs for a directory 
> sub-tree. HCatalog APIs will attempt to build up the directory sub-tree 
> programmatically and runs some code to set the ACLs to match the parent 
> directory.
> Some incorrect assumptions were made when implementing the extended ACLs 
> support. Refer to https://issues.apache.org/jira/browse/HDFS-4685 for the 
> design documents of extended ACLs in HDFS. These documents model the 
> implementation after the POSIX implementation on Linux, which can be found at 
> http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html.
> The code for setting extended ACLs via HCatalog APIs is found in 
> HdfsUtils.java:
> {code}
> if (aclEnabled) {
>   aclStatus =  sourceStatus.getAclStatus();
>   if (aclStatus != 

[jira] [Commented] (HIVE-13989) Extended ACLs are not handled according to specification

[Chris Drome (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133742#comment-16133742
 ] 

Chris Drome commented on HIVE-13989:


For the tests that failed (as opposed to those that timed out), I reran on our 
dev hardware.
I wanted to see if the failure was reproducable and if it also failed at the 
2.2.1 fork point.

|| Test || branch-2.2.1 fork () || branch-2.2 HEAD + HIVE-13989 ||
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_globallimit] | 
FAILED | FAILED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl] | 
PASSED | PASSED |
| 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_1]
 | FAILED | FAILED |
| 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_compact_binary_search]
 | PASSED | PASSED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[selectindate] | PASSED 
| PASSED |
| org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_fast_stats] | 
FAILED | FAILED |
| 
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
| FAILED | FAILED |
| org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
| PASSED | PASSED |
| org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel | 
PASSED | PASSED |
| 
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
 | PASSED | PASSED |
| org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] | PASSED | 
PASSED |
| org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] | PASSED | 
PASSED |

Based on this, HIVE-13989 doesn't appear to be responsible for any of these 
failures.

> Extended ACLs are not handled according to specification
> 
>
> Key: HIVE-13989
> URL: https://issues.apache.org/jira/browse/HIVE-13989
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-13989.1-branch-1.patch, HIVE-13989.1.patch, 
> HIVE-13989.4-branch-2.2.patch, HIVE-13989-branch-1.patch, 
> HIVE-13989-branch-2.2.patch, HIVE-13989-branch-2.2.patch, 
> HIVE-13989-branch-2.2.patch
>
>
> Hive takes two approaches to working with extended ACLs depending on whether 
> data is being produced via a Hive query or HCatalog APIs. A Hive query will 
> run an FsShell command to recursively set the extended ACLs for a directory 
> sub-tree. HCatalog APIs will attempt to build up the directory sub-tree 
> programmatically and runs some code to set the ACLs to match the parent 
> directory.
> Some incorrect assumptions were made when implementing the extended ACLs 
> support. Refer to https://issues.apache.org/jira/browse/HDFS-4685 for the 
> design documents of extended ACLs in HDFS. These documents model the 
> implementation after the POSIX implementation on Linux, which can be found at 
> http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html.
> The code for setting extended ACLs via HCatalog APIs is found in 
> HdfsUtils.java:
> {code}
> if (aclEnabled) {
>   aclStatus =  sourceStatus.getAclStatus();
>   if (aclStatus != null) {
> LOG.trace(aclStatus.toString());
> aclEntries = aclStatus.getEntries();
> removeBaseAclEntries(aclEntries);
> //the ACL api's also expect the tradition user/group/other permission 
> in the form of ACL
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER, 
> sourcePerm.getUserAction()));
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP, 
> sourcePerm.getGroupAction()));
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER, 
> sourcePerm.getOtherAction()));
>   }
> }
> {code}
> We found that DEFAULT extended ACL rules were not being inherited properly by 
> the directory sub-tree, so the above code is incomplete because it 
> effectively drops the DEFAULT rules. The second problem is with the call to 
> {{sourcePerm.getGroupAction()}}, which is incorrect in the case of extended 
> ACLs. When extended ACLs are used the GROUP permission is replaced with the 
> extended ACL mask. So the above code will apply the wrong permissions to the 
> GROUP. Instead the correct GROUP permissions now need to be pulled from the 
> AclEntry as returned by {{getAclStatus().getEntries()}}. See the 
> implementation of the new method {{getDefaultAclEntries}} for details.
> Similar issues exist with the HCatalog API. None of the API accounts for 
> setting extended ACLs on the directory sub-tree. The changes to the HCatalog 
> API allow the extended ACLs to be passed into the required methods similar to 
> how basic permissions are passed in. When building the directory sub-tree the 
> 

[jira] [Updated] (HIVE-17343) create a mechanism to get rid of some globals in HS2

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17343:

Attachment: HIVE-17343.01.patch

Updated the patch.

[~vgumashta] can you take a look? Thanks

> create a mechanism to get rid of some globals in HS2
> 
>
> Key: HIVE-17343
> URL: https://issues.apache.org/jira/browse/HIVE-17343
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17343.01.patch, HIVE-17343.patch
>
>
> The intent is to initialize things once in HS2 ctor/init, and then be able to 
> access them from queries, etc. without using globals or threadlocals.
> Things like future workload management work, LLAP coordinator, materialized 
> view registry, etc. could be accessed this way.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17354) Fix "alter view" for incremental replication

[Tao Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133586#comment-16133586
 ] 

Tao Li commented on HIVE-17354:
---

All test failures are due to flaky tests tracked in HIVE-15058.

[~daijy] Can you please take a look at this change? Thanks!

> Fix "alter view" for incremental replication
> 
>
> Key: HIVE-17354
> URL: https://issues.apache.org/jira/browse/HIVE-17354
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17354.1.patch
>
>
> There is a bug that "alter view" operation is resulting in a view creation 
> operation instead of a overwriting/replacement operation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17241) Change metastore classes to not use the shims

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133571#comment-16133571
 ] 

Alan Gates commented on HIVE-17241:
---

Vaibhav, thanks for the review.  I don't think the test failures are related.  
Most of those have been failing off and on lately (see HIVE-17325).  

> Change metastore classes to not use the shims
> -
>
> Key: HIVE-17241
> URL: https://issues.apache.org/jira/browse/HIVE-17241
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17241.2.patch, HIVE-17241.patch
>
>
> As part of moving the metastore into a standalone package, it will no longer 
> have access to the shims.  This means we need to either copy them or access 
> the underlying Hadoop operations directly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17357) Similar to HIVE-17336, plugin jars are not properly added

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133537#comment-16133537
 ] 

Hive QA commented on HIVE-17357:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882617/HIVE-17357.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10981 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] 
(batchId=9)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6460/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6460/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6460/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882617 - PreCommit-HIVE-Build

> Similar to HIVE-17336, plugin jars are not properly added
> -
>
> Key: HIVE-17357
> URL: https://issues.apache.org/jira/browse/HIVE-17357
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17357.1.patch
>
>
> I forgot to include the same change for LocalHiveSparkClient.java in 
> HIVE-17336. We need to make the same change as HIVE-17336 in 
> LocalHiveSparkClient class to include plugin jars. Maybe we should have a 
> common base class for both LocalHiveSparkClient and RemoteHiveSparkClient to 
> have some common functions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17351) use new slider package installation command in run.sh

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17351:

   Resolution: Fixed
Fix Version/s: 2.4.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Committed to branches. Thanks for the reviews!

> use new slider package installation command in run.sh
> -
>
> Key: HIVE-17351
> URL: https://issues.apache.org/jira/browse/HIVE-17351
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17351.patch
>
>
> The old syntax does not include some perf improvements in newer versions of 
> Slider.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133458#comment-16133458
 ] 

Sergey Shelukhin commented on HIVE-17344:
-

I don't think there's actually a bug since everywhere this buffer is passed in 
it's either allocated exactly to size, or read from disk the same way. Reads 
duplicate it so remaining will not change.
Looking at capacity is not correct in all cases either, e.g. if someone were to 
cache multiple slices of the same large buffer, like LLAP allocator does in 
other places, it would be accounted for multiple times. So, I dunno if this 
change is needed. Perhaps instead at caching time it should check capacity and 
remaining are the same.

> LocalCache element memory usage is not calculated properly.
> ---
>
> Key: HIVE-17344
> URL: https://issues.apache.org/jira/browse/HIVE-17344
> Project: Hive
>  Issue Type: Bug
>Reporter: Janos Gub
>Assignee: Janos Gub
> Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
>   return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads 
> (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus 
> allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think 
> the correct solution would be bb.capacity.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16886) HMS log notifications may have duplicated event IDs if multiple HMS are running concurrently

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133437#comment-16133437
 ] 

Thejas M Nair commented on HIVE-16886:
--

I think due to the flaws with current EVENT_ID, the NL_ID would be right value 
(from db) to use. Longer term we should just have one value and remove the 
EVENT_ID sourced information.
There are two possible approaches to switch to the new 'fixed' value.
One is to introduce a new NL_ID field at api level (maybe EVENT_ID_V2 for 
clarity ?) and eventually drop EVENT_ID field from API. Other option is to just 
switch EVENT_ID field to values sourced from NL_ID. With second approach, no 
changes to applications are needed for the deprecation. Applications would get 
a bump in the event id numbers due to it being sourced from NL_ID which would 
be >= EVENT_ID nos (due to the duplicate issue).

[~spena] What are your thoughts ? 


> HMS log notifications may have duplicated event IDs if multiple HMS are 
> running concurrently
> 
>
> Key: HIVE-16886
> URL: https://issues.apache.org/jira/browse/HIVE-16886
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Metastore
>Reporter: Sergio Peña
>Assignee: anishek
>
> When running multiple Hive Metastore servers and DB notifications are 
> enabled, I could see that notifications can be persisted with a duplicated 
> event ID. 
> This does not happen when running multiple threads in a single HMS node due 
> to the locking acquired on the DbNotificationsLog class, but multiple HMS 
> could cause conflicts.
> The issue is in the ObjectStore#addNotificationEvent() method. The event ID 
> fetched from the datastore is used for the new notification, incremented in 
> the server itself, then persisted or updated back to the datastore. If 2 
> servers read the same ID, then these 2 servers write a new notification with 
> the same ID.
> The event ID is not unique nor a primary key.
> Here's a test case using the TestObjectStore class that confirms this issue:
> {noformat}
> @Test
>   public void testConcurrentAddNotifications() throws ExecutionException, 
> InterruptedException {
> final int NUM_THREADS = 2;
> CountDownLatch countIn = new CountDownLatch(NUM_THREADS);
> CountDownLatch countOut = new CountDownLatch(1);
> HiveConf conf = new HiveConf();
> conf.setVar(HiveConf.ConfVars.METASTORE_EXPRESSION_PROXY_CLASS, 
> MockPartitionExpressionProxy.class.getName());
> ExecutorService executorService = 
> Executors.newFixedThreadPool(NUM_THREADS);
> FutureTask tasks[] = new FutureTask[NUM_THREADS];
> for (int i=0; i   final int n = i;
>   tasks[i] = new FutureTask(new Callable() {
> @Override
> public Void call() throws Exception {
>   ObjectStore store = new ObjectStore();
>   store.setConf(conf);
>   NotificationEvent dbEvent =
>   new NotificationEvent(0, 0, 
> EventMessage.EventType.CREATE_DATABASE.toString(), "CREATE DATABASE DB" + n);
>   System.out.println("ADDING NOTIFICATION");
>   countIn.countDown();
>   countOut.await();
>   store.addNotificationEvent(dbEvent);
>   System.out.println("FINISH NOTIFICATION");
>   return null;
> }
>   });
>   executorService.execute(tasks[i]);
> }
> countIn.await();
> countOut.countDown();
> for (int i = 0; i < NUM_THREADS; ++i) {
>   tasks[i].get();
> }
> NotificationEventResponse eventResponse = 
> objectStore.getNextNotification(new NotificationEventRequest());
> Assert.assertEquals(2, eventResponse.getEventsSize());
> Assert.assertEquals(1, eventResponse.getEvents().get(0).getEventId());
> // This fails because the next notification has an event ID = 1
> Assert.assertEquals(2, eventResponse.getEvents().get(1).getEventId());
>   }
> {noformat}
> The last assertion fails expecting an event ID 1 instead of 2. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17241) Change metastore classes to not use the shims

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133431#comment-16133431
 ] 

Vaibhav Gumashta commented on HIVE-17241:
-

+1 for the changes. Not sure of some of the test failures are related (can't 
access the latest results). 

> Change metastore classes to not use the shims
> -
>
> Key: HIVE-17241
> URL: https://issues.apache.org/jira/browse/HIVE-17241
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17241.2.patch, HIVE-17241.patch
>
>
> As part of moving the metastore into a standalone package, it will no longer 
> have access to the shims.  This means we need to either copy them or access 
> the underlying Hadoop operations directly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133417#comment-16133417
 ] 

Peter Vary commented on HIVE-17292:
---

Thanks [~lirui], and [~xuefuz] for the help and the patience too :)

> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured 
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch, 
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch, HIVE-17292.7.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test 
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster 
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would 
> like to use only 512MB. We should change the fairscheduler configuration to 
> use only the requested 512MB



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17300) WebUI query plan graphs

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133414#comment-16133414
 ] 

Peter Vary commented on HIVE-17300:
---

Thanks [~xuefuz]!

> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
>  Issue Type: Improvement
>  Components: Web UI
>Reporter: Karen Coppage
>Assignee: Karen Coppage
> Attachments: complete_success.png, full_mapred_stats.png, 
> graph_with_mapred_stats.png, HIVE-17300.3.patch, HIVE-17300.patch, 
> last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png
>
>
> Hi all,
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide 
> the option to display the query plan as a nice graph (scroll down for 
> screenshots). If you click on one of the graph’s stages, the plan for that 
> stage appears as text below. 
> Stages are color-coded if they have a status (Success, Error, Running), and 
> the rest are grayed out. Coloring is based on status already available in the 
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This 
> includes the job’s ID, tracking URL (where the logs are found), and mapper 
> and reducer numbers/progress, among other info. 
> The library I’m using for the graph is called vis.js (http://visjs.org/). It 
> has an Apache license, and the only necessary file to be included from this 
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken 
> care of by the client. Plans with more than a given number of stages 
> (default: 25) won't be displayed in order to preserve resources.
> I’d love to hear any and all input from the community about this feature: do 
> you think it’s useful, and is there anything important I’m missing?
> Thanks,
> Karen Coppage
> Review request: https://reviews.apache.org/r/61663/
> Any input is welcome!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16886) HMS log notifications may have duplicated event IDs if multiple HMS are running concurrently

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133389#comment-16133389
 ] 

Sergio Peña commented on HIVE-16886:


[~anishek] I think we should keep the EVENT_ID as it is for now, and can add a 
new variable to the MNotificationLog that tracks the NL_ID. Then when getting 
new notifications, we can check for the NL_ID if the new variable has a value 
or fall back to the EVENT_ID if not. This way we can keep the compatibility 
with current clients using these notifications.

Btw, in the Sentry side, we're going to request HMS notifications in a time 
window so that we can fetch these duplicated events IDs and reapply them in the 
client. Currently, HMS guarantees that IDs are committed in order but 
duplicated events could exist and committed later if an HMS server has delayed 
the transaction commit on the DB. 

> HMS log notifications may have duplicated event IDs if multiple HMS are 
> running concurrently
> 
>
> Key: HIVE-16886
> URL: https://issues.apache.org/jira/browse/HIVE-16886
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Metastore
>Reporter: Sergio Peña
>Assignee: anishek
>
> When running multiple Hive Metastore servers and DB notifications are 
> enabled, I could see that notifications can be persisted with a duplicated 
> event ID. 
> This does not happen when running multiple threads in a single HMS node due 
> to the locking acquired on the DbNotificationsLog class, but multiple HMS 
> could cause conflicts.
> The issue is in the ObjectStore#addNotificationEvent() method. The event ID 
> fetched from the datastore is used for the new notification, incremented in 
> the server itself, then persisted or updated back to the datastore. If 2 
> servers read the same ID, then these 2 servers write a new notification with 
> the same ID.
> The event ID is not unique nor a primary key.
> Here's a test case using the TestObjectStore class that confirms this issue:
> {noformat}
> @Test
>   public void testConcurrentAddNotifications() throws ExecutionException, 
> InterruptedException {
> final int NUM_THREADS = 2;
> CountDownLatch countIn = new CountDownLatch(NUM_THREADS);
> CountDownLatch countOut = new CountDownLatch(1);
> HiveConf conf = new HiveConf();
> conf.setVar(HiveConf.ConfVars.METASTORE_EXPRESSION_PROXY_CLASS, 
> MockPartitionExpressionProxy.class.getName());
> ExecutorService executorService = 
> Executors.newFixedThreadPool(NUM_THREADS);
> FutureTask tasks[] = new FutureTask[NUM_THREADS];
> for (int i=0; i   final int n = i;
>   tasks[i] = new FutureTask(new Callable() {
> @Override
> public Void call() throws Exception {
>   ObjectStore store = new ObjectStore();
>   store.setConf(conf);
>   NotificationEvent dbEvent =
>   new NotificationEvent(0, 0, 
> EventMessage.EventType.CREATE_DATABASE.toString(), "CREATE DATABASE DB" + n);
>   System.out.println("ADDING NOTIFICATION");
>   countIn.countDown();
>   countOut.await();
>   store.addNotificationEvent(dbEvent);
>   System.out.println("FINISH NOTIFICATION");
>   return null;
> }
>   });
>   executorService.execute(tasks[i]);
> }
> countIn.await();
> countOut.countDown();
> for (int i = 0; i < NUM_THREADS; ++i) {
>   tasks[i].get();
> }
> NotificationEventResponse eventResponse = 
> objectStore.getNextNotification(new NotificationEventRequest());
> Assert.assertEquals(2, eventResponse.getEventsSize());
> Assert.assertEquals(1, eventResponse.getEvents().get(0).getEventId());
> // This fails because the next notification has an event ID = 1
> Assert.assertEquals(2, eventResponse.getEvents().get(1).getEventId());
>   }
> {noformat}
> The last assertion fails expecting an event ID 1 instead of 2. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17356) Missing ASF headers 3 classes

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133376#comment-16133376
 ] 

Hive QA commented on HIVE-17356:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882591/HIVE-17356.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10981 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6459/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6459/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6459/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882591 - PreCommit-HIVE-Build

> Missing ASF headers 3 classes
> -
>
> Key: HIVE-17356
> URL: https://issues.apache.org/jira/browse/HIVE-17356
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Trivial
> Attachments: HIVE-17356.01.patch
>
>
> JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java 
> are missing the ASF header that should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17300) WebUI query plan graphs

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133368#comment-16133368
 ] 

Xuefu Zhang commented on HIVE-17300:


I'd like to review it, but haven't got a chance to do so. I will try to find 
time in the coming week. Thanks.

> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
>  Issue Type: Improvement
>  Components: Web UI
>Reporter: Karen Coppage
>Assignee: Karen Coppage
> Attachments: complete_success.png, full_mapred_stats.png, 
> graph_with_mapred_stats.png, HIVE-17300.3.patch, HIVE-17300.patch, 
> last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png
>
>
> Hi all,
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide 
> the option to display the query plan as a nice graph (scroll down for 
> screenshots). If you click on one of the graph’s stages, the plan for that 
> stage appears as text below. 
> Stages are color-coded if they have a status (Success, Error, Running), and 
> the rest are grayed out. Coloring is based on status already available in the 
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This 
> includes the job’s ID, tracking URL (where the logs are found), and mapper 
> and reducer numbers/progress, among other info. 
> The library I’m using for the graph is called vis.js (http://visjs.org/). It 
> has an Apache license, and the only necessary file to be included from this 
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken 
> care of by the client. Plans with more than a given number of stages 
> (default: 25) won't be displayed in order to preserve resources.
> I’d love to hear any and all input from the community about this feature: do 
> you think it’s useful, and is there anything important I’m missing?
> Thanks,
> Karen Coppage
> Review request: https://reviews.apache.org/r/61663/
> Any input is welcome!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17300) WebUI query plan graphs

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133325#comment-16133325
 ] 

Peter Vary commented on HIVE-17300:
---

I have already reviewed the patch before it was submitted, do anyone else would 
like to review it too?

I would appreciate if someone else would review it also!

Thanks,
Peter

> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
>  Issue Type: Improvement
>  Components: Web UI
>Reporter: Karen Coppage
>Assignee: Karen Coppage
> Attachments: complete_success.png, full_mapred_stats.png, 
> graph_with_mapred_stats.png, HIVE-17300.3.patch, HIVE-17300.patch, 
> last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png
>
>
> Hi all,
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide 
> the option to display the query plan as a nice graph (scroll down for 
> screenshots). If you click on one of the graph’s stages, the plan for that 
> stage appears as text below. 
> Stages are color-coded if they have a status (Success, Error, Running), and 
> the rest are grayed out. Coloring is based on status already available in the 
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This 
> includes the job’s ID, tracking URL (where the logs are found), and mapper 
> and reducer numbers/progress, among other info. 
> The library I’m using for the graph is called vis.js (http://visjs.org/). It 
> has an Apache license, and the only necessary file to be included from this 
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken 
> care of by the client. Plans with more than a given number of stages 
> (default: 25) won't be displayed in order to preserve resources.
> I’d love to hear any and all input from the community about this feature: do 
> you think it’s useful, and is there anything important I’m missing?
> Thanks,
> Karen Coppage
> Review request: https://reviews.apache.org/r/61663/
> Any input is welcome!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133322#comment-16133322
 ] 

Peter Vary commented on HIVE-17319:
---

+1 LGTM

> Make BoneCp configurable using hive properties in hive-site.xml
> ---
>
> Key: HIVE-17319
> URL: https://issues.apache.org/jira/browse/HIVE-17319
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17319.01.patch, HIVE-17319.draft.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-17198:
--
Resolution: Cannot Reproduce
Status: Resolved  (was: Patch Available)

With the current code, we are not able to reproduce the issue, so closing the 
ticket.

[~janulatha]: Next time please record a little more information about the 
failure, like:
- hive.log
- exception stack trace
- link to the failed build
- etc.

Thanks,
Peter

> Flaky test: TestBeeLineDriver [smb_mapjoin_7]
> -
>
> Key: HIVE-17198
> URL: https://issues.apache.org/jira/browse/HIVE-17198
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janaki Lahorani
>Assignee: Peter Vary
> Attachments: HIVE-17198.2.patch, HIVE-17198.patch
>
>
> Error:
> Exception running or analyzing the results of the query file: 
> org.apache.hive.beeline.QFile@4f7b68ad



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17357) Similar to HIVE-17336, plugin jars are not properly added

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133292#comment-16133292
 ] 

Xuefu Zhang commented on HIVE-17357:


+1 pending on tests.

> Similar to HIVE-17336, plugin jars are not properly added
> -
>
> Key: HIVE-17357
> URL: https://issues.apache.org/jira/browse/HIVE-17357
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17357.1.patch
>
>
> I forgot to include the same change for LocalHiveSparkClient.java in 
> HIVE-17336. We need to make the same change as HIVE-17336 in 
> LocalHiveSparkClient class to include plugin jars. Maybe we should have a 
> common base class for both LocalHiveSparkClient and RemoteHiveSparkClient to 
> have some common functions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133290#comment-16133290
 ] 

Xuefu Zhang commented on HIVE-15104:


[~lirui], I found it difficulty to backport HIVE-17114 to our code base, so I 
had to give up. However, since you have a configuration to turn this on/off, I 
think it's find to have this and postpone the verification on my side later 
until we upgrade our Hive.

I need some time to review your latest patch as it's different from the 
previous has some low-level class/jar manipulations.

> Hive on Spark generate more shuffle data than hive on mr
> 
>
> Key: HIVE-15104
> URL: https://issues.apache.org/jira/browse/HIVE-15104
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Rui Li
> Attachments: HIVE-15104.1.patch, HIVE-15104.2.patch, 
> HIVE-15104.3.patch, HIVE-15104.4.patch, TPC-H 100G.xlsx
>
>
> the same sql,  running on spark  and mr engine, will generate different size 
> of shuffle data.
> i think it is because of hive on mr just serialize part of HiveKey, but hive 
> on spark which using kryo will serialize full of Hivekey object.  
> what is your opionion?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133272#comment-16133272
 ] 

Hive QA commented on HIVE-17198:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882431/HIVE-17198.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10981 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6458/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6458/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6458/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882431 - PreCommit-HIVE-Build

> Flaky test: TestBeeLineDriver [smb_mapjoin_7]
> -
>
> Key: HIVE-17198
> URL: https://issues.apache.org/jira/browse/HIVE-17198
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janaki Lahorani
>Assignee: Peter Vary
> Attachments: HIVE-17198.2.patch, HIVE-17198.patch
>
>
> Error:
> Exception running or analyzing the results of the query file: 
> org.apache.hive.beeline.QFile@4f7b68ad



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17357) Similar to HIVE-17336, plugin jars are not properly added

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-17357:

Status: Patch Available  (was: Open)

> Similar to HIVE-17336, plugin jars are not properly added
> -
>
> Key: HIVE-17357
> URL: https://issues.apache.org/jira/browse/HIVE-17357
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17357.1.patch
>
>
> I forgot to include the same change for LocalHiveSparkClient.java in 
> HIVE-17336. We need to make the same change as HIVE-17336 in 
> LocalHiveSparkClient class to include plugin jars. Maybe we should have a 
> common base class for both LocalHiveSparkClient and RemoteHiveSparkClient to 
> have some common functions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17357) Similar to HIVE-17336, plugin jars are not properly added

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-17357:

Attachment: HIVE-17357.1.patch

> Similar to HIVE-17336, plugin jars are not properly added
> -
>
> Key: HIVE-17357
> URL: https://issues.apache.org/jira/browse/HIVE-17357
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17357.1.patch
>
>
> I forgot to include the same change for LocalHiveSparkClient.java in 
> HIVE-17336. We need to make the same change as HIVE-17336 in 
> LocalHiveSparkClient class to include plugin jars. Maybe we should have a 
> common base class for both LocalHiveSparkClient and RemoteHiveSparkClient to 
> have some common functions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17357) Similar to HIVE-17336, plugin jars are not properly added

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-17357:
---


> Similar to HIVE-17336, plugin jars are not properly added
> -
>
> Key: HIVE-17357
> URL: https://issues.apache.org/jira/browse/HIVE-17357
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> I forgot to include the same change for LocalHiveSparkClient.java in 
> HIVE-17336. We need to make the same change as HIVE-17336 in 
> LocalHiveSparkClient class to include plugin jars. Maybe we should have a 
> common base class for both LocalHiveSparkClient and RemoteHiveSparkClient to 
> have some common functions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17356) Missing ASF headers 3 classes

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133219#comment-16133219
 ] 

Daniel Dai commented on HIVE-17356:
---

+1

> Missing ASF headers 3 classes
> -
>
> Key: HIVE-17356
> URL: https://issues.apache.org/jira/browse/HIVE-17356
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Trivial
> Attachments: HIVE-17356.01.patch
>
>
> JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java 
> are missing the ASF header that should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133167#comment-16133167
 ] 

Hive QA commented on HIVE-17319:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882587/HIVE-17319.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10986 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6457/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6457/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6457/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882587 - PreCommit-HIVE-Build

> Make BoneCp configurable using hive properties in hive-site.xml
> ---
>
> Key: HIVE-17319
> URL: https://issues.apache.org/jira/browse/HIVE-17319
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17319.01.patch, HIVE-17319.draft.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17356) Missing ASF headers 3 classes

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133117#comment-16133117
 ] 

Eugene Koifman commented on HIVE-17356:
---

+1

> Missing ASF headers 3 classes
> -
>
> Key: HIVE-17356
> URL: https://issues.apache.org/jira/browse/HIVE-17356
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Trivial
> Attachments: HIVE-17356.01.patch
>
>
> JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java 
> are missing the ASF header that should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17356) Missing ASF headers 3 classes

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-17356:
---
Status: Patch Available  (was: Open)

> Missing ASF headers 3 classes
> -
>
> Key: HIVE-17356
> URL: https://issues.apache.org/jira/browse/HIVE-17356
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Trivial
> Attachments: HIVE-17356.01.patch
>
>
> JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java 
> are missing the ASF header that should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17356) Missing ASF headers 3 classes

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-17356:
---
Attachment: HIVE-17356.01.patch

[~daijy], [~ekoifman], [~anishek]
While running a Yetus check for a different patch I ran into some files missing 
the ASF headers authored and/or committed by you.
If you don't mind I would add the headers, please -1 in case you would like to 
retain ownership of the files.

> Missing ASF headers 3 classes
> -
>
> Key: HIVE-17356
> URL: https://issues.apache.org/jira/browse/HIVE-17356
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Trivial
> Attachments: HIVE-17356.01.patch
>
>
> JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java 
> are missing the ASF header that should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17356) Missing ASF headers 3 classes

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara reassigned HIVE-17356:
--


> Missing ASF headers 3 classes
> -
>
> Key: HIVE-17356
> URL: https://issues.apache.org/jira/browse/HIVE-17356
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Trivial
>
> JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java 
> are missing the ASF header that should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-17319:
---
Attachment: HIVE-17319.01.patch

> Make BoneCp configurable using hive properties in hive-site.xml
> ---
>
> Key: HIVE-17319
> URL: https://issues.apache.org/jira/browse/HIVE-17319
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17319.01.patch, HIVE-17319.draft.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-17319:
---
Status: Patch Available  (was: In Progress)

> Make BoneCp configurable using hive properties in hive-site.xml
> ---
>
> Key: HIVE-17319
> URL: https://issues.apache.org/jira/browse/HIVE-17319
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-17319.01.patch, HIVE-17319.draft.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132116#comment-16132116
 ] 

Hive QA commented on HIVE-17292:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882556/HIVE-17292.7.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10981 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=100)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6456/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6456/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6456/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882556 - PreCommit-HIVE-Build

> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured 
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch, 
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch, HIVE-17292.7.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test 
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster 
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would 
> like to use only 512MB. We should change the fairscheduler configuration to 
> use only the requested 512MB



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132072#comment-16132072
 ] 

Hive QA commented on HIVE-17198:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882431/HIVE-17198.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10977 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout 
(batchId=228)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6455/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6455/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6455/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882431 - PreCommit-HIVE-Build

> Flaky test: TestBeeLineDriver [smb_mapjoin_7]
> -
>
> Key: HIVE-17198
> URL: https://issues.apache.org/jira/browse/HIVE-17198
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Janaki Lahorani
>Assignee: Peter Vary
> Attachments: HIVE-17198.2.patch, HIVE-17198.patch
>
>
> Error:
> Exception running or analyzing the results of the query file: 
> org.apache.hive.beeline.QFile@4f7b68ad



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15756) Update/deletes on ACID table throws ArrayIndexOutOfBoundsException

[Ankur varshney (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132063#comment-16132063
 ] 

Ankur varshney commented on HIVE-15756:
---

Hi Team,

Please let us the work around for this issue.

Hive Vesion:-hive-2.1
I can see error in logs ,But in yarn it is showing Succeeded. Please let us
know why it is showing succeeded if it has error in yarn logs.

Error Details:-

2017-08-18 01:43:23,877 [ERROR] [TezChild] |tez.ReduceRecordSource|:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{"reducesinkkey0":{"transactionid":0,"bucketid":-1,"rowid":0}},"value":null}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at
org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:779)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
... 17 more

Thanks
Ankur


> Update/deletes on ACID table throws ArrayIndexOutOfBoundsException
> --
>
> Key: HIVE-15756
> URL: https://issues.apache.org/jira/browse/HIVE-15756
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Kavan Suresh
>Assignee: Eugene Koifman
>Priority: Critical
>
> Update and delete queries on ACID tables fail throwing 
> ArrayIndexOutOfBoundsException.
> {noformat}
> hive> update customer_acid set c_comment = 'foo bar' where c_custkey % 100 = 
> 1;
> Query ID = cstm-hdfs_20170128005823_efa1cdb7-2ad2-4371-ac80-0e35868ad17c
> Total jobs = 1
> Launching Job 1 out of 1
> Tez session was closed. Reopening...
> Session re-established.
> Status: Running (Executing on YARN cluster with App id 
> application_1485331877667_0036)
> 
> VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  
> KILLED
> 
> Map 1 ..   SUCCEEDED 14 1400   0  
>  0
> Reducer 2 FAILED  1  001   1  
>  0
> 
> VERTICES: 01/02  [>>--] 93%   ELAPSED TIME: 23.68 s   
>  
> 
> Status: Failed
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1485331877667_0036_1_01, 
> diagnostics=[Task failed, taskId=task_1485331877667_0036_1_01_00, 
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row (tag=0) 
> {"key":{"reducesinkkey0":{"transactionid":72,"bucketid":1,"rowid":0}},"value":{"_col0":103601,"_col1":"Customer#000103601","_col2":"3cYSrJtAA36vth35
>  
> 

[jira] [Commented] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132057#comment-16132057
 ] 

Rui Li commented on HIVE-17292:
---

+1

> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured 
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch, 
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch, HIVE-17292.7.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test 
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster 
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would 
> like to use only 512MB. We should change the fairscheduler configuration to 
> use only the requested 512MB



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-17292:
--
Attachment: HIVE-17292.7.patch

Rebased the patch

> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured 
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch, 
> HIVE-17292.3.patch, HIVE-17292.5.patch, HIVE-17292.6.patch, HIVE-17292.7.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test 
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster 
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would 
> like to use only 512MB. We should change the fairscheduler configuration to 
> use only the requested 512MB



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17267) Make HMS Notification Listeners typesafe

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-17267:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master.
Thanks for the patch [~zsombor.klara]!

> Make HMS Notification Listeners typesafe
> 
>
> Key: HIVE-17267
> URL: https://issues.apache.org/jira/browse/HIVE-17267
> Project: Hive
>  Issue Type: Bug
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0
>
> Attachments: HIVE-17267.01.patch, HIVE-17267.02.patch, 
> HIVE-17267.03.patch
>
>
> Currently in the HMS we support two types of notification listeners, 
> transactional and non-transactional ones. Transactional listeners will only 
> be invoked if the jdbc transaction finished successfully while 
> non-transactional ones are supposed to be resilient and will be invoked in 
> any case, even for failures.
> Having the same type for these two is a source of confusion and opens the 
> door for misconfigurations. We should try to fix this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17316) Use String.startsWith for the hidden configuration variables

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-17316:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master.
Thanks for the contribution [~zsombor.klara]!

> Use String.startsWith for the hidden configuration variables
> 
>
> Key: HIVE-17316
> URL: https://issues.apache.org/jira/browse/HIVE-17316
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0
>
> Attachments: HIVE-17316.01.patch, HIVE-17316.02.patch, 
> HIVE-17316.03.patch
>
>
> Currently HiveConf variables which should not be displayed to the user need 
> to be enumerated. We should enhance this to be able to hide configuration 
> variables by string prefix not just full equality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17355) Casting to Decimal along with UNION ALL gives incosistent results

[Aditya Allamraju (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Allamraju updated HIVE-17355:

Description: 
Extra trailing zeros are added when running "union all" on the tables 
containing decimal data types.

*Version:* Hive 2.1

*Steps to repro:-*

{code:java}

1) CREATE TABLE `decisample`(
  `a` decimal(8,2),
  `b` int,
  `c` decimal(5,2))
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'maprfs:/user/hive/warehouse/decisample'

2) CREATE TABLE `decisample3`(
  `a` decimal(8,2),
  `b` int,
  `c` decimal(5,2))
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'maprfs:/user/hive/warehouse/decisample3'

3)hive> select * from decisample3;
OK
1.002   3.00
7.008   9.00

4)hive> select * from decisample;
OK
4.005   6.00

5) query:- 
select a1.a, '' as a1b,'' as a1c from decisample a1 union all select 
a2.a,a2.b,a2.c from decisample3 a2;

o/p:-
OK
4.00NULL
1.002   3.00
7.008   9.00
Time taken: 87.993 seconds, Fetched: 3 row(s)

6)select a2.a,a2.b,a2.c from decisample3 a2 union all select a1.a, '' as a1b,'' 
as a1c from decisample a1;

o/p:-
4.00
1.002   3
7.008   9

{code}

Steps 5 is yielding 18 trailing zeros where as step 6 query is yieldings no 
trailing  zero.

Observation:

1. Hive is trying to run the UNION ALL after ensuring the SELECT's are 
semantically same(equal number of columns and same datatypes). To do this, it 
is implicitly type casting the values where required.

>From the explain plan, type casting is not consistent when done 2 different 
>ways:
a)  select-1  UNION ALL select-2 (Query-5 in above comment)
vs
b) select-2 UNION ALL select-2   (Query-6 in above comment)

Showing only the "expresssions" part of execution plans

Query-5:


{code:java}
..
..
Map Operator Tree:
  TableScan
alias: a1
Statistics: Num rows: 1 Data size: 11 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), '' (type: string), null 
(type: decimal(38,18))
  outputColumnNames: _col0, _col1, _col2
..
..
TableScan
alias: a2
Statistics: Num rows: 2 Data size: 22 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), UDFToString(b) (type: 
string), CAST( c AS decimal(38,18)) (type: decimal(38,18))


{code}

Query-6:


{code:java}
..
..
Map Operator Tree:
  TableScan
alias: a2
Statistics: Num rows: 2 Data size: 22 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), UDFToString(b) (type: 
string), UDFToString(c) (type: string)
..
..
TableScan
alias: a1
Statistics: Num rows: 1 Data size: 11 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), '' (type: string), '' (type: 
string)
..
..


{code}
Attaching the execution plans for both queries for reference.

2. The reason for 18 zeros in query-5 above is due to casting NULL to Decimal.
And by default, the precision and scale are taken as (38,18) in Hive. This 
could be the reason for 18 zeros.

3. This is repeating every time implicit type casting is happening on EMPTY 
strings.

If excluding few columns in one of the SELECT statement is absolutely 
necessary, then the only Workaround is to explicitly type cast the empty 
strings to same Datatypes as the Other Select statement which included the 
columns.


For ex:
Q1:

select a,b,c from decisample3
union all
select a,cast(' ' as int),cast(' ' as decimal) from decisample;

Q2:

select a,cast(' ' as int),cast(' ' as decimal) from decisample
union all
select a,b,c from decisample3;

Both the above queries will give consistent result now.

cast(' ' as int) ---> this was cast to INT, same as datatype of "b"
cast(' ' as decimal) ---> this was cast to decimal, same as datatype of "c"

  was:
Extra trailing zeros are added when running "union all" on the tables 
containing decimal data types.

Version: Hive 2.1

Steps to repro:-


1) CREATE TABLE `decisample`(
  `a` decimal(8,2),
  `b` int,
  `c` decimal(5,2))
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  

[jira] [Assigned] (HIVE-17355) Casting to Decimal along with UNION ALL gives incosistent results

[Aditya Allamraju (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Allamraju reassigned HIVE-17355:
---

Assignee: Aditya Allamraju

> Casting to Decimal along with UNION ALL gives incosistent results
> -
>
> Key: HIVE-17355
> URL: https://issues.apache.org/jira/browse/HIVE-17355
> Project: Hive
>  Issue Type: Bug
>  Components: Parser, UDF
>Affects Versions: 2.1.0, 2.1.1
> Environment: CentOS 7.2
>Reporter: Aditya Allamraju
>Assignee: Aditya Allamraju
>
> Extra trailing zeros are added when running "union all" on the tables 
> containing decimal data types.
> Version: Hive 2.1
> Steps to repro:-
> 
> 1) CREATE TABLE `decisample`(
>   `a` decimal(8,2),
>   `b` int,
>   `c` decimal(5,2))
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'maprfs:/user/hive/warehouse/decisample'
> 2) CREATE TABLE `decisample3`(
>   `a` decimal(8,2),
>   `b` int,
>   `c` decimal(5,2))
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'maprfs:/user/hive/warehouse/decisample3'
> 3)hive> select * from decisample3;
> OK
> 1.002   3.00
> 7.008   9.00
> 4)hive> select * from decisample;
> OK
> 4.005   6.00
> 5) query:- 
> select a1.a, '' as a1b,'' as a1c from decisample a1 union all select 
> a2.a,a2.b,a2.c from decisample3 a2;
> o/p:-
> OK
> 4.00NULL
> 1.002   3.00
> 7.008   9.00
> Time taken: 87.993 seconds, Fetched: 3 row(s)
> 6)select a2.a,a2.b,a2.c from decisample3 a2 union all select a1.a, '' as 
> a1b,'' as a1c from decisample a1;
> o/p:-
> 4.00
> 1.002   3
> 7.008   9
> Steps 5 is yielding 18 trailing zeros where as step 6 query is yieldings no 
> trailing  zero.
> Observation:
> 1. Hive is trying to run the UNION ALL after ensuring the SELECT's are 
> semantically same(equal number of columns and same datatypes). To do this, it 
> is implicitly type casting the values where required.
> From the explain plan, type casting is not consistent when done 2 different 
> ways:
> a)  select-1  UNION ALL select-2 (Query-5 in above comment)
> vs
> b) select-2 UNION ALL select-2   (Query-6 in above comment)
> Showing only the "expresssions" part of execution plans
> Query-5:
> 
> ..
> ..
> Map Operator Tree:
>   TableScan
> alias: a1
> Statistics: Num rows: 1 Data size: 11 Basic stats: COMPLETE 
> Column stats: NONE
> Select Operator
>   expressions: a (type: decimal(8,2)), '' (type: string), null 
> (type: decimal(38,18))
>   outputColumnNames: _col0, _col1, _col2
> ..
> ..
> TableScan
> alias: a2
> Statistics: Num rows: 2 Data size: 22 Basic stats: COMPLETE 
> Column stats: NONE
> Select Operator
>   expressions: a (type: decimal(8,2)), UDFToString(b) (type: 
> string), CAST( c AS decimal(38,18)) (type: decimal(38,18))
> Query-6:
> 
> ..
> ..
> Map Operator Tree:
>   TableScan
> alias: a2
> Statistics: Num rows: 2 Data size: 22 Basic stats: COMPLETE 
> Column stats: NONE
> Select Operator
>   expressions: a (type: decimal(8,2)), UDFToString(b) (type: 
> string), UDFToString(c) (type: string)
> ..
> ..
> TableScan
> alias: a1
> Statistics: Num rows: 1 Data size: 11 Basic stats: COMPLETE 
> Column stats: NONE
> Select Operator
>   expressions: a (type: decimal(8,2)), '' (type: string), '' 
> (type: string)
> ..
> ..
> Attaching the execution plans for both queries for reference.
> 2. The reason for 18 zeros in query-5 above is due to casting NULL to Decimal.
> And by default, the precision and scale are taken as (38,18) in Hive. This 
> could be the reason for 18 zeros.
> 3. This is repeating every time implicit type casting is happening on EMPTY 
> strings.
> If excluding few columns in one of the SELECT statement is absolutely 
> necessary, then the only Workaround is to explicitly type cast the empty 
> strings to same Datatypes as the Other Select statement which included the 
> columns.
> For ex:
> Q1:
> select a,b,c from decisample3
> union all
> select a,cast(' ' as int),cast(' ' as decimal) from decisample;
> Q2:
> select a,cast(' ' as int),cast(' ' as decimal) from decisample
> union all
> select a,b,c from decisample3;
> Both the above 

[jira] [Updated] (HIVE-17355) Casting to Decimal along with UNION ALL gives incosistent results

[Aditya Allamraju (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Allamraju updated HIVE-17355:

Description: 
Extra trailing zeros are added when running "union all" on the tables 
containing decimal data types.

Version: Hive 2.1

Steps to repro:-


1) CREATE TABLE `decisample`(
  `a` decimal(8,2),
  `b` int,
  `c` decimal(5,2))
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'maprfs:/user/hive/warehouse/decisample'

2) CREATE TABLE `decisample3`(
  `a` decimal(8,2),
  `b` int,
  `c` decimal(5,2))
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'maprfs:/user/hive/warehouse/decisample3'

3)hive> select * from decisample3;
OK
1.002   3.00
7.008   9.00

4)hive> select * from decisample;
OK
4.005   6.00

5) query:- 
select a1.a, '' as a1b,'' as a1c from decisample a1 union all select 
a2.a,a2.b,a2.c from decisample3 a2;

o/p:-
OK
4.00NULL
1.002   3.00
7.008   9.00
Time taken: 87.993 seconds, Fetched: 3 row(s)

6)select a2.a,a2.b,a2.c from decisample3 a2 union all select a1.a, '' as a1b,'' 
as a1c from decisample a1;

o/p:-
4.00
1.002   3
7.008   9

Steps 5 is yielding 18 trailing zeros where as step 6 query is yieldings no 
trailing  zero.

Observation:

1. Hive is trying to run the UNION ALL after ensuring the SELECT's are 
semantically same(equal number of columns and same datatypes). To do this, it 
is implicitly type casting the values where required.

>From the explain plan, type casting is not consistent when done 2 different 
>ways:
a)  select-1  UNION ALL select-2 (Query-5 in above comment)
vs
b) select-2 UNION ALL select-2   (Query-6 in above comment)

Showing only the "expresssions" part of execution plans

Query-5:

..
..
Map Operator Tree:
  TableScan
alias: a1
Statistics: Num rows: 1 Data size: 11 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), '' (type: string), null 
(type: decimal(38,18))
  outputColumnNames: _col0, _col1, _col2
..
..
TableScan
alias: a2
Statistics: Num rows: 2 Data size: 22 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), UDFToString(b) (type: 
string), CAST( c AS decimal(38,18)) (type: decimal(38,18))


Query-6:

..
..
Map Operator Tree:
  TableScan
alias: a2
Statistics: Num rows: 2 Data size: 22 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), UDFToString(b) (type: 
string), UDFToString(c) (type: string)
..
..
TableScan
alias: a1
Statistics: Num rows: 1 Data size: 11 Basic stats: COMPLETE Column 
stats: NONE
Select Operator
  expressions: a (type: decimal(8,2)), '' (type: string), '' (type: 
string)
..
..

Attaching the execution plans for both queries for reference.

2. The reason for 18 zeros in query-5 above is due to casting NULL to Decimal.
And by default, the precision and scale are taken as (38,18) in Hive. This 
could be the reason for 18 zeros.

3. This is repeating every time implicit type casting is happening on EMPTY 
strings.

If excluding few columns in one of the SELECT statement is absolutely 
necessary, then the only Workaround is to explicitly type cast the empty 
strings to same Datatypes as the Other Select statement which included the 
columns.


For ex:
Q1:

select a,b,c from decisample3
union all
select a,cast(' ' as int),cast(' ' as decimal) from decisample;

Q2:

select a,cast(' ' as int),cast(' ' as decimal) from decisample
union all
select a,b,c from decisample3;

Both the above queries will give consistent result now.

cast(' ' as int) ---> this was cast to INT, same as datatype of "b"
cast(' ' as decimal) ---> this was cast to decimal, same as datatype of "c"

  was:
Extra trailing zeros are added when running "union all" on the tables 
containing decimal data types.

Env:-
=
mapr-hivemetastore-2.1.201703241741-1.noarch
mapr-hive-2.1.201703241741-1.noarch
mapr-hiveserver2-2.1.201703241741-1.noarch

Steps to repro:-


1) CREATE TABLE `decisample`(
  `a` decimal(8,2),
  `b` int,
  `c` decimal(5,2))
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  

[jira] [Commented] (HIVE-17277) HiveMetastoreClient Log name is wrong

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132014#comment-16132014
 ] 

Hive QA commented on HIVE-17277:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882538/HIVE-17277.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10977 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6454/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6454/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6454/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882538 - PreCommit-HIVE-Build

> HiveMetastoreClient Log name is wrong
> -
>
> Key: HIVE-17277
> URL: https://issues.apache.org/jira/browse/HIVE-17277
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Minor
> Attachments: HIVE-17277.2.patch, HIVE-17277.3.patch, HIVE-17277.patch
>
>
> The name of Log for HiveMetastoreClient is "hive.metastore". It's confused 
> for users to trace hive log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-14747:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to the master.
Thanks [~zsombor.klara]!

We did not yet activated it on the ptest server, since there are local changes 
there, and we would like clean up them before changing anything.

> Remove JAVA paths from profiles by sending them from ptest-client
> -
>
> Key: HIVE-14747
> URL: https://issues.apache.org/jira/browse/HIVE-14747
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive, Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Fix For: 3.0.0
>
> Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, 
> HIVE-14747.03.patch, HIVE-14747.04.patch, HIVE-14747.05.patch, 
> HIVE-14747.06.patch
>
>
> Hive ptest uses some properties files per branch that contain information 
> about how to execute the tests.
> This profile includes JAVA paths to build and execute the tests. We should 
> get rid of these by passing such information from Jenkins to the 
> ptest-server. In case a profile needs a different java version, then we can 
> create a specific Jenkins job for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17277) HiveMetastoreClient Log name is wrong

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara reassigned HIVE-17277:
--

Assignee: Zac Zhou  (was: Barna Zsombor Klara)

> HiveMetastoreClient Log name is wrong
> -
>
> Key: HIVE-17277
> URL: https://issues.apache.org/jira/browse/HIVE-17277
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Minor
> Attachments: HIVE-17277.2.patch, HIVE-17277.3.patch, HIVE-17277.patch
>
>
> The name of Log for HiveMetastoreClient is "hive.metastore". It's confused 
> for users to trace hive log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17277) HiveMetastoreClient Log name is wrong

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-17277:
---
Attachment: HIVE-17277.3.patch

[~yuan_zac]
The PTest framework was restarted while your tests were running so the results 
will not be updated in this Jira.
I reuploaded the same patch (HIVE-17277.2.patch) to retrigger the testing for 
you.
Sorry for the inconvenience.

> HiveMetastoreClient Log name is wrong
> -
>
> Key: HIVE-17277
> URL: https://issues.apache.org/jira/browse/HIVE-17277
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Zac Zhou
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17277.2.patch, HIVE-17277.3.patch, HIVE-17277.patch
>
>
> The name of Log for HiveMetastoreClient is "hive.metastore". It's confused 
> for users to trace hive log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17277) HiveMetastoreClient Log name is wrong

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara reassigned HIVE-17277:
--

Assignee: Barna Zsombor Klara  (was: Zac Zhou)

> HiveMetastoreClient Log name is wrong
> -
>
> Key: HIVE-17277
> URL: https://issues.apache.org/jira/browse/HIVE-17277
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Zac Zhou
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17277.2.patch, HIVE-17277.patch
>
>
> The name of Log for HiveMetastoreClient is "hive.metastore". It's confused 
> for users to trace hive log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-13989) Extended ACLs are not handled according to specification

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131895#comment-16131895
 ] 

Hive QA commented on HIVE-13989:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882512/HIVE-13989.4-branch-2.2.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 9934 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=244)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out) 
(batchId=225)
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=167)
[acid_globallimit.q,alter_merge_2_orc.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=173)

[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,orc_merge6.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join3.q,index_bitmap_auto.q,vector_outer_join2.q,vector_outer_join1.q,orc_merge1.q,orc_merge_diff_fs.q,load_hdfs_file_with_space_in_the_name.q,scriptfile1_win.q,quotedid_smb.q,truncate_column_buckets.q,orc_merge3.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=174)

[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,insert_overwrite_directory2.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,vector_outer_join0.q,orc_merge4.q,temp_table_external.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_semijoin.q,auto_sortmerge_join_16.q,schemeAuthority.q,index_bitmap3.q,external_table_with_space_in_location_path.q,parallel_orderby.q,infer_bucket_sort_map_operators.q,bucketizedhiveinputformat.q,remote_script.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=175)

[scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,bucket4.q,input16_cc.q,bucket5.q,infer_bucket_sort_merge.q,constprog_partitioner.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,load_fs2.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,vector_inner_join.q,orc_merge7.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=118)

[bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=119)

[join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,stats16.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=120)

[parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,groupby3.q,bucketsortoptimize_insert_8.q,union20.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=121)

[ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,order2.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=122)

[groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,auto_join11.q,load_dyn_part7.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=123)

[skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,join_reorder.q,sample1.q,bucketmapjoin8.q,union34.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=124)

[avro_joins.q,skewjoinopt16.q,auto_join14.q,vectorization_14.q,auto_join26.q,stats1.q,cbo_stats.q,auto_sortmerge_join_6.q,union22.q,union_remove_24.q,union_view.q,smb_mapjoin_22.q,stats15.q,ptf_matchpath.q,transform_ppr1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=125)

[jira] [Updated] (HIVE-17194) JDBC: Implement Gzip compression for HTTP mode

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-17194:
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master, thanks [~thejas]

> JDBC: Implement Gzip compression for HTTP mode
> --
>
> Key: HIVE-17194
> URL: https://issues.apache.org/jira/browse/HIVE-17194
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2, JDBC
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>  Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-17194.1.patch, HIVE-17194.2.patch, 
> HIVE-17194.3.patch
>
>
> {code}
> POST /cliservice HTTP/1.1
> Content-Type: application/x-thrift
> Accept: application/x-thrift
> User-Agent: Java/THttpClient/HC
> Authorization: Basic YW5vbnltb3VzOmFub255bW91cw==
> Content-Length: 71
> Host: localhost:10007
> Connection: Keep-Alive
> Accept-Encoding: gzip,deflate
> X-XSRF-HEADER: true
> {code}
> The Beeline client clearly sends out HTTP compression headers which are 
> ignored by the HTTP service layer in HS2.
> After patch, result looks like
> {code}
> HTTP/1.1 200 OK
> Date: Tue, 01 Aug 2017 01:47:23 GMT
> Content-Type: application/x-thrift
> Vary: Accept-Encoding, User-Agent
> Content-Encoding: gzip
> Transfer-Encoding: chunked
> Server: Jetty(9.3.8.v20160314)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17194) JDBC: Implement Gzip compression for HTTP mode

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-17194:
---
Labels: TODOC3.0  (was: )

> JDBC: Implement Gzip compression for HTTP mode
> --
>
> Key: HIVE-17194
> URL: https://issues.apache.org/jira/browse/HIVE-17194
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2, JDBC
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>  Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-17194.1.patch, HIVE-17194.2.patch, 
> HIVE-17194.3.patch
>
>
> {code}
> POST /cliservice HTTP/1.1
> Content-Type: application/x-thrift
> Accept: application/x-thrift
> User-Agent: Java/THttpClient/HC
> Authorization: Basic YW5vbnltb3VzOmFub255bW91cw==
> Content-Length: 71
> Host: localhost:10007
> Connection: Keep-Alive
> Accept-Encoding: gzip,deflate
> X-XSRF-HEADER: true
> {code}
> The Beeline client clearly sends out HTTP compression headers which are 
> ignored by the HTTP service layer in HS2.
> After patch, result looks like
> {code}
> HTTP/1.1 200 OK
> Date: Tue, 01 Aug 2017 01:47:23 GMT
> Content-Type: application/x-thrift
> Vary: Accept-Encoding, User-Agent
> Content-Encoding: gzip
> Transfer-Encoding: chunked
> Server: Jetty(9.3.8.v20160314)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-16948:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks [~kellyzly] for the work.

> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Fix For: 3.0.0
>
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch, 
> HIVE-16948.5.patch, HIVE-16948.6.patch, HIVE-16948.7.patch, HIVE-16948.patch
>
>
> in 
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
>  in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all 
> select distinct(ds) as ds from srcpart) s where s.ds in (select 
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain 
> {code}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-1 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Spark
>   Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
>   DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
>   Vertices:
> Map 10 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: max(ds)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Reduce Output Operator
> sort order: 
> Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: min(ds)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Reduce Output Operator
> sort order: 
> Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Reducer 11 
> Reduce Operator Tree:
>   Group By Operator
> aggregations: max(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE 
> Column stats: NONE
> Filter Operator
>   predicate: _col0 is not null (type: boolean)
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats: 
> COMPLETE Column stats: NONE
> Select Operator
>   expressions: _col0 (type: string)
>   outputColumnNames: _col0
>   Statistics: Num rows: 2 Data size: 368 Basic stats: 
> 

[jira] [Commented] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark

[liyunzhang_intel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131842#comment-16131842
 ] 

liyunzhang_intel commented on HIVE-16948:
-

thanks for your review, [~lirui],[~stakiar],[~pvary]!

> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch, 
> HIVE-16948.5.patch, HIVE-16948.6.patch, HIVE-16948.7.patch, HIVE-16948.patch
>
>
> in 
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
>  in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all 
> select distinct(ds) as ds from srcpart) s where s.ds in (select 
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain 
> {code}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-1 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Spark
>   Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
>   DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
>   Vertices:
> Map 10 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: max(ds)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Reduce Output Operator
> sort order: 
> Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: min(ds)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Reduce Output Operator
> sort order: 
> Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Reducer 11 
> Reduce Operator Tree:
>   Group By Operator
> aggregations: max(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE 
> Column stats: NONE
> Filter Operator
>   predicate: _col0 is not null (type: boolean)
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats: 
> COMPLETE Column stats: NONE
> Select Operator
>   expressions: _col0 (type: string)
>   outputColumnNames: _col0
>   Statistics: Num rows: 2 Data size: 368 Basic stats: 
> COMPLETE Column stats: NONE
>   Group By Operator
>  

[jira] [Commented] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131834#comment-16131834
 ] 

Rui Li commented on HIVE-16948:
---

+1. Thanks for the update [~kellyzly]

> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch, 
> HIVE-16948.5.patch, HIVE-16948.6.patch, HIVE-16948.7.patch, HIVE-16948.patch
>
>
> in 
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
>  in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all 
> select distinct(ds) as ds from srcpart) s where s.ds in (select 
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain 
> {code}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-1 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Spark
>   Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
>   DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
>   Vertices:
> Map 10 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: max(ds)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Reduce Output Operator
> sort order: 
> Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: min(ds)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Reduce Output Operator
> sort order: 
> Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Reducer 11 
> Reduce Operator Tree:
>   Group By Operator
> aggregations: max(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE 
> Column stats: NONE
> Filter Operator
>   predicate: _col0 is not null (type: boolean)
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Group By Operator
> keys: _col0 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 368 Basic stats: 
> COMPLETE Column stats: NONE
> Select Operator
>   expressions: _col0 (type: string)
>   outputColumnNames: _col0
>   Statistics: Num rows: 2 Data size: 368 Basic stats: 
> COMPLETE Column stats: NONE
>   Group By Operator
> keys: _col0 

[jira] [Updated] (HIVE-17277) HiveMetastoreClient Log name is wrong

[Zac Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zac Zhou updated HIVE-17277:

Attachment: HIVE-17277.2.patch

Fixing the compiling error

> HiveMetastoreClient Log name is wrong
> -
>
> Key: HIVE-17277
> URL: https://issues.apache.org/jira/browse/HIVE-17277
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Minor
> Attachments: HIVE-17277.2.patch, HIVE-17277.patch
>
>
> The name of Log for HiveMetastoreClient is "hive.metastore". It's confused 
> for users to trace hive log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16948) Invalid explain when running dynamic partition pruning query in Hive On Spark

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131822#comment-16131822
 ] 

Hive QA commented on HIVE-16948:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12882506/HIVE-16948.7.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10977 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=180)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=180)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6451/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6451/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6451/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12882506 - PreCommit-HIVE-Build

> Invalid explain when running dynamic partition pruning query in Hive On Spark
> -
>
> Key: HIVE-16948
> URL: https://issues.apache.org/jira/browse/HIVE-16948
> Project: Hive
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16948_1.patch, HIVE-16948.2.patch, 
> HIVE-16948.5.patch, HIVE-16948.6.patch, HIVE-16948.7.patch, HIVE-16948.patch
>
>
> in 
> [union_subquery.q|https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q#L107]
>  in spark_dynamic_partition_pruning.q
> {code}
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.spark.dynamic.partition.pruning=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.strict.checks.cartesian.product=false;
> explain select ds from (select distinct(ds) as ds from srcpart union all 
> select distinct(ds) as ds from srcpart) s where s.ds in (select 
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
> {code}
> explain 
> {code}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-1 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Spark
>   Edges:
> Reducer 11 <- Map 10 (GROUP, 1)
> Reducer 13 <- Map 12 (GROUP, 1)
>   DagName: root_20170622231525_20a777e5-e659-4138-b605-65f8395e18e2:2
>   Vertices:
> Map 10 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: max(ds)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
>   Reduce Output Operator
> sort order: 
> Statistics: Num rows: 1 Data size: 184 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string)
> Map 12 
> Map Operator Tree:
> TableScan
>   alias: srcpart
>   Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: ds
> Statistics: Num rows: 1 Data size: 23248 Basic stats: 
> PARTIAL Column stats: NONE
> Group By Operator
>   aggregations: min(ds)
>   mode: hash
>