[jira] [Updated] (HIVE-19089) Create/Replicate Allocate write-id event

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-19089:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Create/Replicate Allocate write-id event
> 
>
> Key: HIVE-19089
> URL: https://issues.apache.org/jira/browse/HIVE-19089
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19089.01.patch, HIVE-19089.02.patch, 
> HIVE-19089.03.patch, HIVE-19089.04.patch, HIVE-19089.05.patch, 
> HIVE-19089.06.patch, HIVE-19089.07.patch, HIVE-19089.08.patch
>
>
> *EVENT_ALLOCATE_WRITE_ID*
> *Source Warehouse:*
>  * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format 
> etc.
>  * Capture this event when allocate a table write ID from the sequence table 
> by ACID operation.
>  * Repl dump should read this event from EventNotificationTable and dump the 
> message.
> *Target Warehouse:*
>  * Repl load should read the event from the dump and get the message.
>  * Validate if source txn ID from the event is there in the source-target txn 
> ID map. If not there, just noop the event.
>  * If valid, then Allocate table write ID from sequence table
> *Extend listener notify event API to add two new parameter , dbconn and 
> sqlgenerator to add the events to notification_log table within the same 
> transaction* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19089) Create/Replicate Allocate write-id event

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440515#comment-16440515
 ] 

Sankar Hariappan commented on HIVE-19089:
-

08.patch is committed to master!

Thanks [~maheshk114] for the contribution!

> Create/Replicate Allocate write-id event
> 
>
> Key: HIVE-19089
> URL: https://issues.apache.org/jira/browse/HIVE-19089
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19089.01.patch, HIVE-19089.02.patch, 
> HIVE-19089.03.patch, HIVE-19089.04.patch, HIVE-19089.05.patch, 
> HIVE-19089.06.patch, HIVE-19089.07.patch, HIVE-19089.08.patch
>
>
> *EVENT_ALLOCATE_WRITE_ID*
> *Source Warehouse:*
>  * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format 
> etc.
>  * Capture this event when allocate a table write ID from the sequence table 
> by ACID operation.
>  * Repl dump should read this event from EventNotificationTable and dump the 
> message.
> *Target Warehouse:*
>  * Repl load should read the event from the dump and get the message.
>  * Validate if source txn ID from the event is there in the source-target txn 
> ID map. If not there, just noop the event.
>  * If valid, then Allocate table write ID from sequence table
> *Extend listener notify event API to add two new parameter , dbconn and 
> sqlgenerator to add the events to notification_log table within the same 
> transaction* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18946) Fix columnstats merge NPE

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-18946:

   Resolution: Fixed
Fix Version/s: 3.1.0
   3.0.0
   Status: Resolved  (was: Patch Available)

pushed to master and branch-3; thank you [~abstractdog] for fixing this!

> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch, 
> HIVE-18946.03.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main] 
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at 
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at 
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2 
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19104) When test MetaStore is started with retry the instances should be independent

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440459#comment-16440459
 ] 

Hive QA commented on HIVE-19104:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
43s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
21s{color} | {color:red} standalone-metastore: The patch generated 3 new + 264 
unchanged - 4 fixed = 267 total (was 268) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10264/dev-support/hive-personality.sh
 |
| git revision | master / 7fb088b |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10264/yetus/patch-mvninstall-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10264/yetus/diff-checkstyle-standalone-metastore.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10264/yetus/patch-asflicense-problems.txt
 |
| modules | C: hcatalog/core itests/hive-unit itests/util standalone-metastore 
U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10264/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> When test MetaStore is started with retry the instances should be independent
> -
>
> Key: HIVE-19104
> URL: https://issues.apache.org/jira/browse/HIVE-19104
> Project: Hive
>  Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19104.2.patch, HIVE-19104.3.patch, 
> HIVE-19104.4.patch, HIVE-19104.5.patch, HIVE-19104.6.patch, 
> HIVE-19104.7.patch, HIVE-19104.patch
>
>
> When multiple MetaStore instances are started with 
> {{MetaStoreTestUtils.startMetaStoreWithRetry}} currently they use the same 
> JDBC url, and warehouse directory. This can cause problem in the tests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18910:
--
Attachment: HIVE-18910.33.patch

> Migrate to Murmur hash for shuffle and bucketing
> 
>
> Key: HIVE-18910
> URL: https://issues.apache.org/jira/browse/HIVE-18910
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch, 
> HIVE-18910.11.patch, HIVE-18910.12.patch, HIVE-18910.13.patch, 
> HIVE-18910.14.patch, HIVE-18910.15.patch, HIVE-18910.16.patch, 
> HIVE-18910.17.patch, HIVE-18910.18.patch, HIVE-18910.19.patch, 
> HIVE-18910.2.patch, HIVE-18910.20.patch, HIVE-18910.21.patch, 
> HIVE-18910.22.patch, HIVE-18910.23.patch, HIVE-18910.24.patch, 
> HIVE-18910.25.patch, HIVE-18910.26.patch, HIVE-18910.27.patch, 
> HIVE-18910.28.patch, HIVE-18910.29.patch, HIVE-18910.3.patch, 
> HIVE-18910.30.patch, HIVE-18910.31.patch, HIVE-18910.32.patch, 
> HIVE-18910.33.patch, HIVE-18910.4.patch, HIVE-18910.5.patch, 
> HIVE-18910.6.patch, HIVE-18910.7.patch, HIVE-18910.8.patch, HIVE-18910.9.patch
>
>
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19126) CachedStore: Use memory estimation to limit cache size during prewarm

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440614#comment-16440614
 ] 

Hive QA commented on HIVE-19126:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919193/HIVE-19126.6.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 78 failed/errored test(s), 13816 tests 
executed
*Failed tests:*
{noformat}
TestCachedStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=217)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out) 
(batchId=217)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestDeadline - did not produce a TEST-*.xml file (likely timed out) 
(batchId=217)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out) 
(batchId=217)
TestMetaStoreEventListenerOnlyOnCommit - did not produce a TEST-*.xml file 
(likely timed out) (batchId=217)
TestMetaStoreInitListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestMetaStoreListenersError - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestMetaStoreSchemaInfo - did not produce a TEST-*.xml file (likely timed out) 
(batchId=217)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-19125) TestStreaming fails with java.lang.NoClassDefFoundError

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440647#comment-16440647
 ] 

Hive QA commented on HIVE-19125:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m  0s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10266/dev-support/hive-personality.sh
 |
| git revision | master / fa9e743 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10266/yetus/patch-asflicense-problems.txt
 |
| modules | C: hcatalog hcatalog/hcatalog-pig-adapter U: hcatalog |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10266/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> TestStreaming fails with java.lang.NoClassDefFoundError
> ---
>
> Key: HIVE-19125
> URL: https://issues.apache.org/jira/browse/HIVE-19125
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Test
>Affects Versions: 3.0.0
> Environment: $ uname -a
> Linux 90ab83b583eb 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
> $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
>Reporter: Pravin Dsilva
>Priority: Major
> Attachments: HIVE-19125.1.patch, HIVE-19125.2.patch
>
>
> The failures appear only when running the entire test suite but pass when the 
> tests are run individually.
> command used: mvn test -fn -pl ql,hcatalog,hcatalog/streaming 
> -Dtest=TestStreaming
>  
> {code:java}
> [INFO] Results:
> [INFO] 
> [ERROR] Errors: 
> [ERROR]   
> TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
>  ? NoClassDefFound
> [ERROR]   
> TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
>  ? NoClassDefFound
> [ERROR]   
> TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
>  ? NoClassDefFound
> [ERROR]   
> TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
>  ? NoClassDefFound
> [ERROR]   
> TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
>  ? 

[jira] [Updated] (HIVE-19089) Create/Replicate Allocate write-id event

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-19089:
---
Attachment: HIVE-19089.08.patch

> Create/Replicate Allocate write-id event
> 
>
> Key: HIVE-19089
> URL: https://issues.apache.org/jira/browse/HIVE-19089
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19089.01.patch, HIVE-19089.02.patch, 
> HIVE-19089.03.patch, HIVE-19089.04.patch, HIVE-19089.05.patch, 
> HIVE-19089.06.patch, HIVE-19089.07.patch, HIVE-19089.08.patch
>
>
> *EVENT_ALLOCATE_WRITE_ID*
> *Source Warehouse:*
>  * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format 
> etc.
>  * Capture this event when allocate a table write ID from the sequence table 
> by ACID operation.
>  * Repl dump should read this event from EventNotificationTable and dump the 
> message.
> *Target Warehouse:*
>  * Repl load should read the event from the dump and get the message.
>  * Validate if source txn ID from the event is there in the source-target txn 
> ID map. If not there, just noop the event.
>  * If valid, then Allocate table write ID from sequence table
> *Extend listener notify event API to add two new parameter , dbconn and 
> sqlgenerator to add the events to notification_log table within the same 
> transaction* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19089) Create/Replicate Allocate write-id event

[mahesh kumar behera (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440512#comment-16440512
 ] 

mahesh kumar behera commented on HIVE-19089:


All the test failures are tested and verified to be passing in local test . 
Patch HIVE-19089.08.patch is attached for one check style issue fix.

> Create/Replicate Allocate write-id event
> 
>
> Key: HIVE-19089
> URL: https://issues.apache.org/jira/browse/HIVE-19089
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19089.01.patch, HIVE-19089.02.patch, 
> HIVE-19089.03.patch, HIVE-19089.04.patch, HIVE-19089.05.patch, 
> HIVE-19089.06.patch, HIVE-19089.07.patch, HIVE-19089.08.patch
>
>
> *EVENT_ALLOCATE_WRITE_ID*
> *Source Warehouse:*
>  * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format 
> etc.
>  * Capture this event when allocate a table write ID from the sequence table 
> by ACID operation.
>  * Repl dump should read this event from EventNotificationTable and dump the 
> message.
> *Target Warehouse:*
>  * Repl load should read the event from the dump and get the message.
>  * Validate if source txn ID from the event is there in the source-target txn 
> ID map. If not there, just noop the event.
>  * If valid, then Allocate table write ID from sequence table
> *Extend listener notify event API to add two new parameter , dbconn and 
> sqlgenerator to add the events to notification_log table within the same 
> transaction* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19126) CachedStore: Use memory estimation to limit cache size during prewarm

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440544#comment-16440544
 ] 

Hive QA commented on HIVE-19126:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} storage-api: The patch generated 61 new + 0 unchanged 
- 0 fixed = 61 total (was 0) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} llap-server: The patch generated 0 new + 25 
unchanged - 75 fixed = 25 total (was 100) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
19s{color} | {color:red} standalone-metastore: The patch generated 19 new + 149 
unchanged - 3 fixed = 168 total (was 152) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10265/dev-support/hive-personality.sh
 |
| git revision | master / fa9e743 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10265/yetus/diff-checkstyle-storage-api.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10265/yetus/diff-checkstyle-standalone-metastore.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10265/yetus/patch-asflicense-problems.txt
 |
| modules | C: storage-api llap-server standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10265/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> CachedStore: Use memory estimation to limit cache size during prewarm
> -
>
> Key: HIVE-19126
> URL: https://issues.apache.org/jira/browse/HIVE-19126
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19126.1.patch, HIVE-19126.2.patch, 
> HIVE-19126.3.patch, HIVE-19126.4.patch, HIVE-19126.5.patch, 
> HIVE-19126.5.patch, HIVE-19126.6.patch
>
>
> We can rely on 
> 

[jira] [Updated] (HIVE-19220) perfLogger = SessionState.getPerfLogger() in Driver.java need to combine

[shengsiwei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shengsiwei updated HIVE-19220:
--
Status: Open  (was: Patch Available)

> perfLogger = SessionState.getPerfLogger() in Driver.java need to combine
> 
>
> Key: HIVE-19220
> URL: https://issues.apache.org/jira/browse/HIVE-19220
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: shengsiwei
>Assignee: shengsiwei
>Priority: Minor
> Attachments: HIVE-19220.1.patch
>
>
> I find  duplicate code in Driver.java. We need to eliminate duplicate code to 
> improve the simplicity of the code
>  
> Before modification
> {code:java}
> PerfLogger perfLogger = null;
> if (!alreadyCompiled) {
>  // compile internal will automatically reset the perf logger
>  compileInternal(command, true);
>  // then we continue to use this perf logger
>  perfLogger = SessionState.getPerfLogger();
> } else {
>  // reuse existing perf logger.
>  perfLogger = SessionState.getPerfLogger();
>  // Since we're reusing the compiled plan, we need to update its start time 
> for current run
>  plan.setQueryStartTime(perfLogger.getStartTime(PerfLogger.DRIVER_RUN));
> }
> {code}
>  
> After  modification 
>  
> {code:java}
> PerfLogger perfLogger = SessionState.getPerfLogger();
>   if (!alreadyCompiled) {
> // compile internal will automatically reset the perf logger
> compileInternal(command, true);
>   } else {
> // Since we're reusing the compiled plan, we need to update its start 
> time for current run
> 
> plan.setQueryStartTime(perfLogger.getStartTime(PerfLogger.DRIVER_RUN));
>   }
> {code}
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19159) TestMTQueries.testMTQueries1 failure

[Zoltan Haindrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440758#comment-16440758
 ] 

Zoltan Haindrich commented on HIVE-19159:
-

+1 pending tests

> TestMTQueries.testMTQueries1 failure
> 
>
> Key: HIVE-19159
> URL: https://issues.apache.org/jira/browse/HIVE-19159
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Vineet Garg
>Assignee: Laszlo Bodor
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-19159.01.patch
>
>
> I have confirmed that HIVE-18051 caused this failure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19159) TestMTQueries.testMTQueries1 failure

[Laszlo Bodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440754#comment-16440754
 ] 

Laszlo Bodor commented on HIVE-19159:
-

[~vgarg], [~kgyrtkirk] : patch attached, the fix is only 1 line 
(qt.startSessionState(false)), all others are method extractions in QTestUtil.

> TestMTQueries.testMTQueries1 failure
> 
>
> Key: HIVE-19159
> URL: https://issues.apache.org/jira/browse/HIVE-19159
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Vineet Garg
>Assignee: Laszlo Bodor
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-19159.01.patch
>
>
> I have confirmed that HIVE-18051 caused this failure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19103) Reading required column only in nested structure schema in ORC

[Ashish Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Sharma updated HIVE-19103:
-
Status: In Progress  (was: Patch Available)

> Reading required column only in nested structure schema in ORC
> --
>
> Key: HIVE-19103
> URL: https://issues.apache.org/jira/browse/HIVE-19103
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-19103-0.patch, HIVE-19103-1.patch
>
>
> Reading required columns only in nested structure schema
> Example - 
> *Current state* - 
> Schema  -  struct>
> Query - select c.e.f from t where c.e.f > 10;
> Current state - read entire c struct from the file and then filter because 
> "hive.io.file.readcolumn.ids" is referred due to which all the children 
> column are select to read from the file.
> Conf -
>  _hive.io.file.readcolumn.ids  = "2"
>  hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -   
> boolean[ ] include  = [true,false,false,true,true,true,true,true]
> *Expected state* -
> Schema  -  struct>
> Query - select c.e.f from t where c.e.f > 10;
> Expected state - instead of reading entire c struct from the file just read 
> only the f column by referring the  " hive.io.file.readNestedColumn.paths".
> Conf -
>  _hive.io.file.readcolumn.ids  = "2"
>  hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -   
> boolean[ ] include  = [true,false,false,true,false,true,true,false]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-12369) Native Vector GroupBy (Part 1)

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-12369:

Attachment: (was: HIVE-12369.096.patch)

> Native Vector GroupBy (Part 1)
> --
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, 
> HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, 
> HIVE-12369.094.patch, HIVE-12369.095.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed 
> for Native Vector MapJoin, etc.
> Patch is currently limited to a single Long key with a single COUNT 
> aggregation.  Or, a single Long key and no aggregation also known as 
> duplicate reduction.
> 3 new classes introduces that stored the count in the slot table and don't 
> allocate hash elements:
> {noformat}
>   COUNT(column)  VectorGroupByHashLongKeyCountColumnOperator  
>   COUNT(key) VectorGroupByHashLongKeyCountKeyOperator
>   COUNT(*)   VectorGroupByHashLongKeyCountStarOperator   
> {noformat}
> And the duplicate reduction operator a single Long key:
> {noformat}
>   VectorGroupByHashLongKeyDuplicateReductionOperator
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-12369) Native Vector GroupBy (Part 1)

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-12369:

Attachment: HIVE-12369.096.patch

> Native Vector GroupBy (Part 1)
> --
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, 
> HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, 
> HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed 
> for Native Vector MapJoin, etc.
> Patch is currently limited to a single Long key with a single COUNT 
> aggregation.  Or, a single Long key and no aggregation also known as 
> duplicate reduction.
> 3 new classes introduces that stored the count in the slot table and don't 
> allocate hash elements:
> {noformat}
>   COUNT(column)  VectorGroupByHashLongKeyCountColumnOperator  
>   COUNT(key) VectorGroupByHashLongKeyCountKeyOperator
>   COUNT(*)   VectorGroupByHashLongKeyCountStarOperator   
> {noformat}
> And the duplicate reduction operator a single Long key:
> {noformat}
>   VectorGroupByHashLongKeyDuplicateReductionOperator
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19158) Fix NPE in the HiveMetastore add partition tests

[Marta Kuczora (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marta Kuczora updated HIVE-19158:
-
Attachment: HIVE-19158.2.patch

> Fix NPE in the HiveMetastore add partition tests
> 
>
> Key: HIVE-19158
> URL: https://issues.apache.org/jira/browse/HIVE-19158
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Test
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
> Attachments: HIVE-19158.1.patch, HIVE-19158.2.patch
>
>
> The TestAddPartitions and TestAddPartitionsFromPartSpec tests revealed that 
> NPE is thrown in some cases. These NPEs could be prevented with a simple null 
> check and a MetaException with a proper error message should be thrown 
> instead.
>  Example: NPE is thrown in the following test cases
>  * TestAddPartitions
>  ** testAddPartitionNullPartition
>  ** testAddPartitionNullValue
>  ** testAddPartitionsNullList
>  * TestAddPartitionsFromPartSpec
>  ** testAddPartitionSpecNullSpec
>  ** testAddPartitionSpecNullPartList
>  ** testAddPartitionSpecNoDB
>  ** testAddPartitionSpecNoTable
>  ** testAddPartitionSpecNoDBAndTableInPartition
>  ** testAddPartitionSpecNullPart
>  ** testAddPartitionSpecChangeRootPathToNull
>  ** testAddPartitionSpecWithSharedSDNullSd
>  ** testAddPartitionSpecWithSharedSDNoValue
>  ** testAddPartitionSpecNullValue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19009) Retain and use runtime statistics during hs2 lifetime

[Zoltan Haindrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440674#comment-16440674
 ] 

Zoltan Haindrich commented on HIVE-19009:
-

updated rb; I've rerun some test - they doesn't seemed to be related - updated 
QTestUtil to clear the stat cache between cases...

> Retain and use runtime statistics during hs2 lifetime
> -
>
> Key: HIVE-19009
> URL: https://issues.apache.org/jira/browse/HIVE-19009
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19009.01.patch, HIVE-19009.02.patch, 
> HIVE-19009.03.patch, HIVE-19009.04.patch, HIVE-19009.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19009) Retain and use runtime statistics during hs2 lifetime

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-19009:

Attachment: HIVE-19009.05.patch

> Retain and use runtime statistics during hs2 lifetime
> -
>
> Key: HIVE-19009
> URL: https://issues.apache.org/jira/browse/HIVE-19009
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19009.01.patch, HIVE-19009.02.patch, 
> HIVE-19009.03.patch, HIVE-19009.04.patch, HIVE-19009.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19226) Extend storage-api to print timestamp values in UTC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-19226:
--


> Extend storage-api to print timestamp values in UTC
> ---
>
> Key: HIVE-19226
> URL: https://issues.apache.org/jira/browse/HIVE-19226
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Related to HIVE-12192. Create new method that prints values in UTC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19103) Reading required column only in nested structure schema in ORC

[Ashish Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Sharma updated HIVE-19103:
-
Fix Version/s: 2.3.2
Affects Version/s: 2.3.2
   Status: Patch Available  (was: In Progress)

> Reading required column only in nested structure schema in ORC
> --
>
> Key: HIVE-19103
> URL: https://issues.apache.org/jira/browse/HIVE-19103
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.3.2
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.3.2
>
> Attachments: HIVE-19103-0.patch, HIVE-19103-1.patch, 
> HIVE-19103-2.patch
>
>
> Reading required columns only in nested structure schema
> Example - 
> *Current state* - 
> Schema  -  struct>
> Query - select c.e.f from t where c.e.f > 10;
> Current state - read entire c struct from the file and then filter because 
> "hive.io.file.readcolumn.ids" is referred due to which all the children 
> column are select to read from the file.
> Conf -
>  _hive.io.file.readcolumn.ids  = "2"
>  hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -   
> boolean[ ] include  = [true,false,false,true,true,true,true,true]
> *Expected state* -
> Schema  -  struct>
> Query - select c.e.f from t where c.e.f > 10;
> Expected state - instead of reading entire c struct from the file just read 
> only the f column by referring the  " hive.io.file.readNestedColumn.paths".
> Conf -
>  _hive.io.file.readcolumn.ids  = "2"
>  hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -   
> boolean[ ] include  = [true,false,false,true,false,true,true,false]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19103) Reading required column only in nested structure schema in ORC

[Ashish Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Sharma updated HIVE-19103:
-
Attachment: HIVE-19103-2.patch

> Reading required column only in nested structure schema in ORC
> --
>
> Key: HIVE-19103
> URL: https://issues.apache.org/jira/browse/HIVE-19103
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-19103-0.patch, HIVE-19103-1.patch, 
> HIVE-19103-2.patch
>
>
> Reading required columns only in nested structure schema
> Example - 
> *Current state* - 
> Schema  -  struct>
> Query - select c.e.f from t where c.e.f > 10;
> Current state - read entire c struct from the file and then filter because 
> "hive.io.file.readcolumn.ids" is referred due to which all the children 
> column are select to read from the file.
> Conf -
>  _hive.io.file.readcolumn.ids  = "2"
>  hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -   
> boolean[ ] include  = [true,false,false,true,true,true,true,true]
> *Expected state* -
> Schema  -  struct>
> Query - select c.e.f from t where c.e.f > 10;
> Expected state - instead of reading entire c struct from the file just read 
> only the f column by referring the  " hive.io.file.readNestedColumn.paths".
> Conf -
>  _hive.io.file.readcolumn.ids  = "2"
>  hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -   
> boolean[ ] include  = [true,false,false,true,false,true,true,false]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440782#comment-16440782
 ] 

Hive QA commented on HIVE-19219:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919243/HIVE-19219.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 87 failed/errored test(s), 13843 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Updated] (HIVE-19226) Extend storage-api to print timestamp values in UTC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19226:
---
Status: Patch Available  (was: In Progress)

> Extend storage-api to print timestamp values in UTC
> ---
>
> Key: HIVE-19226
> URL: https://issues.apache.org/jira/browse/HIVE-19226
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19226.patch
>
>
> Related to HIVE-12192. Create new method that prints values in UTC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19226) Extend storage-api to print timestamp values in UTC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19226:
---
Attachment: HIVE-19226.patch

> Extend storage-api to print timestamp values in UTC
> ---
>
> Key: HIVE-19226
> URL: https://issues.apache.org/jira/browse/HIVE-19226
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19226.patch
>
>
> Related to HIVE-12192. Create new method that prints values in UTC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19158) Fix NPE in the HiveMetastore add partition tests

[Marta Kuczora (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440810#comment-16440810
 ] 

Marta Kuczora commented on HIVE-19158:
--

Reloaded the patch as it was missing from the pre-commit job pool.

> Fix NPE in the HiveMetastore add partition tests
> 
>
> Key: HIVE-19158
> URL: https://issues.apache.org/jira/browse/HIVE-19158
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Test
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
> Attachments: HIVE-19158.1.patch, HIVE-19158.2.patch
>
>
> The TestAddPartitions and TestAddPartitionsFromPartSpec tests revealed that 
> NPE is thrown in some cases. These NPEs could be prevented with a simple null 
> check and a MetaException with a proper error message should be thrown 
> instead.
>  Example: NPE is thrown in the following test cases
>  * TestAddPartitions
>  ** testAddPartitionNullPartition
>  ** testAddPartitionNullValue
>  ** testAddPartitionsNullList
>  * TestAddPartitionsFromPartSpec
>  ** testAddPartitionSpecNullSpec
>  ** testAddPartitionSpecNullPartList
>  ** testAddPartitionSpecNoDB
>  ** testAddPartitionSpecNoTable
>  ** testAddPartitionSpecNoDBAndTableInPartition
>  ** testAddPartitionSpecNullPart
>  ** testAddPartitionSpecChangeRootPathToNull
>  ** testAddPartitionSpecWithSharedSDNullSd
>  ** testAddPartitionSpecWithSharedSDNoValue
>  ** testAddPartitionSpecNullValue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-19226) Extend storage-api to print timestamp values in UTC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-19226 started by Jesus Camacho Rodriguez.
--
> Extend storage-api to print timestamp values in UTC
> ---
>
> Key: HIVE-19226
> URL: https://issues.apache.org/jira/browse/HIVE-19226
> Project: Hive
>  Issue Type: Bug
>  Components: storage-api
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19226.patch
>
>
> Related to HIVE-12192. Create new method that prints values in UTC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440740#comment-16440740
 ] 

Hive QA commented on HIVE-19219:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
23s{color} | {color:red} itests/hive-unit: The patch generated 11 new + 594 
unchanged - 39 fixed = 605 total (was 633) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10269/dev-support/hive-personality.sh
 |
| git revision | master / 5404635 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10269/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10269/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests/hive-unit standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10269/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Incremental REPL DUMP should throw error if requested events are cleaned-up.
> 
>
> Key: HIVE-19219
> URL: https://issues.apache.org/jira/browse/HIVE-19219
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19219.01.patch
>
>
> This is the case where the events were deleted on source because of old event 
> purging and hence min(source event id) > target event id (last replicated 
> event id).
> Repl dump should fail in this case so that user can drop the database and 
> bootstrap again.
> The next incremental repl dump could check if the events fetched from source 
> notification_log table is retrieved in continuous sequence with no events 
> missing. If any event is missing, it should throw error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19159) TestMTQueries.testMTQueries1 failure

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-19159:

Status: Patch Available  (was: Open)

> TestMTQueries.testMTQueries1 failure
> 
>
> Key: HIVE-19159
> URL: https://issues.apache.org/jira/browse/HIVE-19159
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Vineet Garg
>Assignee: Laszlo Bodor
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-19159.01.patch
>
>
> I have confirmed that HIVE-18051 caused this failure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19159) TestMTQueries.testMTQueries1 failure

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-19159:

Attachment: HIVE-19159.01.patch

> TestMTQueries.testMTQueries1 failure
> 
>
> Key: HIVE-19159
> URL: https://issues.apache.org/jira/browse/HIVE-19159
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Vineet Garg
>Assignee: Laszlo Bodor
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-19159.01.patch
>
>
> I have confirmed that HIVE-18051 caused this failure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19222) TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: GC overhead limit exceeded"

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440815#comment-16440815
 ] 

Hive QA commented on HIVE-19222:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10270/dev-support/hive-personality.sh
 |
| git revision | master / 5404635 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10270/yetus/patch-asflicense-problems.txt
 |
| modules | C: . U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10270/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: 
> GC overhead limit exceeded"
> ---
>
> Key: HIVE-19222
> URL: https://issues.apache.org/jira/browse/HIVE-19222
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19222.1.patch
>
>
> TestNegativeCliDriver tests are failing with OOM recently. Not sure why. I 
> will try to increase the memory to test out.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19171) Persist runtime statistics in metastore

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-19171:

Status: Patch Available  (was: Open)

01wip01: includes HIVE-19009 

> Persist runtime statistics in metastore
> ---
>
> Key: HIVE-19171
> URL: https://issues.apache.org/jira/browse/HIVE-19171
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19171.01wip01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19125) TestStreaming fails with java.lang.NoClassDefFoundError

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440717#comment-16440717
 ] 

Hive QA commented on HIVE-19125:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919212/HIVE-19125.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 98 failed/errored test(s), 13841 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Updated] (HIVE-12369) Native Vector GroupBy (Part 1)

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-12369:

Description: 
Implement Native Vector GroupBy using fast hash table technology developed for 
Native Vector MapJoin, etc.

Patch is currently limited to a single key with a single COUNT aggregation.  
Or, a single key and no aggregation also known as duplicate reduction.

3 new kinds of classes introduces that stored the count in the slot table and 
don't allocate hash elements.  Example:
{noformat}
  COUNT(column)  VectorGroupByHashLongKeySingleCountColumnOperator  
  COUNT(key) VectorGroupByHashLongKeySingleCountKeyOperator
  COUNT(*)   VectorGroupByHashLongKeySingleCountStarOperator   
{noformat}
And the duplicate reduction operator a single key.  Example:
{noformat}
  VectorGroupByHashLongKeyDuplicateReductionOperator
{noformat}

  was:
Implement Native Vector GroupBy using fast hash table technology developed for 
Native Vector MapJoin, etc.

Patch is currently limited to a single Long key with a single COUNT 
aggregation.  Or, a single Long key and no aggregation also known as duplicate 
reduction.

3 new classes introduces that stored the count in the slot table and don't 
allocate hash elements:
{noformat}
  COUNT(column)  VectorGroupByHashLongKeyCountColumnOperator  
  COUNT(key) VectorGroupByHashLongKeyCountKeyOperator
  COUNT(*)   VectorGroupByHashLongKeyCountStarOperator   
{noformat}
And the duplicate reduction operator a single Long key:
{noformat}
  VectorGroupByHashLongKeyDuplicateReductionOperator
{noformat}


> Native Vector GroupBy (Part 1)
> --
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, 
> HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, 
> HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed 
> for Native Vector MapJoin, etc.
> Patch is currently limited to a single key with a single COUNT aggregation.  
> Or, a single key and no aggregation also known as duplicate reduction.
> 3 new kinds of classes introduces that stored the count in the slot table and 
> don't allocate hash elements.  Example:
> {noformat}
>   COUNT(column)  VectorGroupByHashLongKeySingleCountColumnOperator  
>   COUNT(key) VectorGroupByHashLongKeySingleCountKeyOperator
>   COUNT(*)   VectorGroupByHashLongKeySingleCountStarOperator   
> {noformat}
> And the duplicate reduction operator a single key.  Example:
> {noformat}
>   VectorGroupByHashLongKeyDuplicateReductionOperator
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19171) Persist runtime statistics in metastore

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-19171:

Attachment: HIVE-19171.01wip01.patch

> Persist runtime statistics in metastore
> ---
>
> Key: HIVE-19171
> URL: https://issues.apache.org/jira/browse/HIVE-19171
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19171.01wip01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19104) When test MetaStore is started with retry the instances should be independent

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440506#comment-16440506
 ] 

Hive QA commented on HIVE-19104:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919178/HIVE-19104.7.patch

{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 166 failed/errored test(s), 14236 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
 (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] 
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_dyn_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_map_operators]
 (batchId=93)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[joinneg] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[parquet_alter_part_table_drop_columns]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ptf_negative_JoinWithAmbigousAlias]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14] 
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subq_insert] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_from]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_exists_implicit_gby]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_lhs] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_on] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notexists_implicit_gby]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_columns]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_distinct2]
 

[jira] [Commented] (HIVE-19222) TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: GC overhead limit exceeded"

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440850#comment-16440850
 ] 

Hive QA commented on HIVE-19222:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919260/HIVE-19222.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 14237 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
 (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] 
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_dyn_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_map_operators]
 (batchId=93)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[joinneg] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ptf_negative_JoinWithAmbigousAlias]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subq_insert] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[update_notnull_constraint]
 (batchId=95)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=225)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitMultiInsertOverwriteTable
 (batchId=261)
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testMetastoreVersion 
(batchId=227)
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionMatching 
(batchId=227)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/10270/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10270/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10270/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12919260 - PreCommit-HIVE-Build

> TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: 
> GC overhead limit exceeded"
> ---
>
> Key: HIVE-19222
> URL: https://issues.apache.org/jira/browse/HIVE-19222
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> 

[jira] [Updated] (HIVE-19171) Persist runtime statistics in metastore

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-19171:

Attachment: HIVE-19171.01wip02.patch

> Persist runtime statistics in metastore
> ---
>
> Key: HIVE-19171
> URL: https://issues.apache.org/jira/browse/HIVE-19171
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19171.01wip01.patch, HIVE-19171.01wip02.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-19219:

Status: Open  (was: Patch Available)

> Incremental REPL DUMP should throw error if requested events are cleaned-up.
> 
>
> Key: HIVE-19219
> URL: https://issues.apache.org/jira/browse/HIVE-19219
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19219.01.patch
>
>
> This is the case where the events were deleted on source because of old event 
> purging and hence min(source event id) > target event id (last replicated 
> event id).
> Repl dump should fail in this case so that user can drop the database and 
> bootstrap again.
> The next incremental repl dump could check if the events fetched from source 
> notification_log table is retrieved in continuous sequence with no events 
> missing. If any event is missing, it should throw error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19186) Multi Table INSERT statements query has a flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440977#comment-16440977
 ] 

Hive QA commented on HIVE-19186:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919259/HIVE-19186.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 13843 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=96)

[jira] [Updated] (HIVE-19046) Refactor the common parts of the HiveMetastore add_partition_core and add_partitions_pspec_core methods

[Marta Kuczora (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marta Kuczora updated HIVE-19046:
-
Status: Patch Available  (was: Open)

> Refactor the common parts of the HiveMetastore add_partition_core and 
> add_partitions_pspec_core methods
> ---
>
> Key: HIVE-19046
> URL: https://issues.apache.org/jira/browse/HIVE-19046
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
> Attachments: HIVE-19046.1.patch
>
>
> This is a follow-up Jira of the 
> [HIVE-18696|https://issues.apache.org/jira/browse/HIVE-18696] 
> [review|https://reviews.apache.org/r/65716/].
> The biggest part of these methods use the same code. It would make sense to 
> move this code part to a common method.
> This code is almost the same in the two methods:
> {code}
> List partFutures = Lists.newArrayList();
> final Table table = tbl;
> for (final Partition part : parts) {
>   if (!part.getTableName().equals(tblName) || 
> !part.getDbName().equals(dbName)) {
> throw new MetaException("Partition does not belong to target 
> table "
> + dbName + "." + tblName + ": " + part);
>   }
>   boolean shouldAdd = startAddPartition(ms, part, ifNotExists);
>   if (!shouldAdd) {
> existingParts.add(part);
> LOG.info("Not adding partition " + part + " as it already 
> exists");
> continue;
>   }
>   final UserGroupInformation ugi;
>   try {
> ugi = UserGroupInformation.getCurrentUser();
>   } catch (IOException e) {
> throw new RuntimeException(e);
>   }
>   partFutures.add(threadPool.submit(new Callable() {
> @Override
> public Partition call() throws Exception {
>   ugi.doAs(new PrivilegedExceptionAction() {
> @Override
> public Object run() throws Exception {
>   try {
> boolean madeDir = createLocationForAddedPartition(table, 
> part);
> if (addedPartitions.put(new PartValEqWrapper(part), 
> madeDir) != null) {
>   // Technically, for ifNotExists case, we could insert 
> one and discard the other
>   // because the first one now "exists", but it seems 
> better to report the problem
>   // upstream as such a command doesn't make sense.
>   throw new MetaException("Duplicate partitions in the 
> list: " + part);
> }
> initializeAddedPartition(table, part, madeDir);
>   } catch (MetaException e) {
> throw new IOException(e.getMessage(), e);
>   }
>   return null;
> }
>   });
>   return part;
> }
>   }));
> }
> try {
>   for (Future partFuture : partFutures) {
> Partition part = partFuture.get();
> if (part != null) {
>   newParts.add(part);
> }
>   }
> } catch (InterruptedException | ExecutionException e) {
>   // cancel other tasks
>   for (Future partFuture : partFutures) {
> partFuture.cancel(true);
>   }
>   throw new MetaException(e.getMessage());
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19046) Refactor the common parts of the HiveMetastore add_partition_core and add_partitions_pspec_core methods

[Marta Kuczora (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marta Kuczora updated HIVE-19046:
-
Attachment: HIVE-19046.1.patch

> Refactor the common parts of the HiveMetastore add_partition_core and 
> add_partitions_pspec_core methods
> ---
>
> Key: HIVE-19046
> URL: https://issues.apache.org/jira/browse/HIVE-19046
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
> Attachments: HIVE-19046.1.patch
>
>
> This is a follow-up Jira of the 
> [HIVE-18696|https://issues.apache.org/jira/browse/HIVE-18696] 
> [review|https://reviews.apache.org/r/65716/].
> The biggest part of these methods use the same code. It would make sense to 
> move this code part to a common method.
> This code is almost the same in the two methods:
> {code}
> List partFutures = Lists.newArrayList();
> final Table table = tbl;
> for (final Partition part : parts) {
>   if (!part.getTableName().equals(tblName) || 
> !part.getDbName().equals(dbName)) {
> throw new MetaException("Partition does not belong to target 
> table "
> + dbName + "." + tblName + ": " + part);
>   }
>   boolean shouldAdd = startAddPartition(ms, part, ifNotExists);
>   if (!shouldAdd) {
> existingParts.add(part);
> LOG.info("Not adding partition " + part + " as it already 
> exists");
> continue;
>   }
>   final UserGroupInformation ugi;
>   try {
> ugi = UserGroupInformation.getCurrentUser();
>   } catch (IOException e) {
> throw new RuntimeException(e);
>   }
>   partFutures.add(threadPool.submit(new Callable() {
> @Override
> public Partition call() throws Exception {
>   ugi.doAs(new PrivilegedExceptionAction() {
> @Override
> public Object run() throws Exception {
>   try {
> boolean madeDir = createLocationForAddedPartition(table, 
> part);
> if (addedPartitions.put(new PartValEqWrapper(part), 
> madeDir) != null) {
>   // Technically, for ifNotExists case, we could insert 
> one and discard the other
>   // because the first one now "exists", but it seems 
> better to report the problem
>   // upstream as such a command doesn't make sense.
>   throw new MetaException("Duplicate partitions in the 
> list: " + part);
> }
> initializeAddedPartition(table, part, madeDir);
>   } catch (MetaException e) {
> throw new IOException(e.getMessage(), e);
>   }
>   return null;
> }
>   });
>   return part;
> }
>   }));
> }
> try {
>   for (Future partFuture : partFutures) {
> Partition part = partFuture.get();
> if (part != null) {
>   newParts.add(part);
> }
>   }
> } catch (InterruptedException | ExecutionException e) {
>   // cancel other tasks
>   for (Future partFuture : partFutures) {
> partFuture.cancel(true);
>   }
>   throw new MetaException(e.getMessage());
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19164) TestMetastoreVersion failures

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441005#comment-16441005
 ] 

Hive QA commented on HIVE-19164:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10272/dev-support/hive-personality.sh
 |
| git revision | master / 5404635 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10272/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10272/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Vineet Garg
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.02.patch, HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws 
> an exception {{No matching version found}}. 
> To fetch schema version this method executes {code:sql} SELECT FROM 
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever 
> reason execution returns empty result set resulting in the exception. Both 
> test failures are due to this exception. I tried debugging the query 
> execution but didn't really go nowhere with it. I suspect this could be due 
> to recent metastore changes. I tried reproducing this outside test but with 
> no success.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-19219:

Status: Patch Available  (was: Open)

Attached 02.patch with change in logic
 * If last replicated event is missing from NOTIFICATION_LOG table, then shall 
assume some events might be missing from dump and throw error.
 * Validate this after dumping all the events to confirm if events not deleted 
concurrently when dumping in progress.
 * No constraint to get events in sequence.

Request [~maheshk114] to review it!

> Incremental REPL DUMP should throw error if requested events are cleaned-up.
> 
>
> Key: HIVE-19219
> URL: https://issues.apache.org/jira/browse/HIVE-19219
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19219.01.patch, HIVE-19219.02.patch
>
>
> This is the case where the events were deleted on source because of old event 
> purging and hence min(source event id) > target event id (last replicated 
> event id).
> Repl dump should fail in this case so that user can drop the database and 
> bootstrap again.
> Cleaner thread is concurrently removing the expired events from 
> NOTIFICATION_LOG table. So, it is necessary to check if the current dump 
> missed any event while dumping. It is guaranteed that cleaner would clean 
> sequentially based on event id/eventTime. So, it is enough to check if the 
> last replicated event is still there in NOTIFICATION_LOG table even after 
> dumping all events.
> If it is not found, then likely some events missed in the dump and hence 
> throw error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-19219:

Description: 
This is the case where the events were deleted on source because of old event 
purging and hence min(source event id) > target event id (last replicated event 
id).

Repl dump should fail in this case so that user can drop the database and 
bootstrap again.

Cleaner thread is concurrently removing the expired events from 
NOTIFICATION_LOG table. So, it is necessary to check if the current dump missed 
any event while dumping. It is guaranteed that cleaner would clean sequentially 
based on event id/eventTime. So, it is enough to check if the last replicated 
event is still there in NOTIFICATION_LOG table even after dumping all events.

If it is not found, then likely some events missed in the dump and hence throw 
error.

  was:
This is the case where the events were deleted on source because of old event 
purging and hence min(source event id) > target event id (last replicated event 
id).

Repl dump should fail in this case so that user can drop the database and 
bootstrap again.

Cleaner thread is concurrently removing the expired events from 
NOTIFICATION_LOG table. So, it is necessary to check if the current dump missed 
any event while dumping. Also, it is guaranteed that cleaner would clean 
sequentially based on event id. So, it is enough to check if the first event > 
last event id is still there in NOTIFICATION_LOG table even after dumping all 
events.

If it is not found, then likely some events missed in the dump and hence throw 
error.


> Incremental REPL DUMP should throw error if requested events are cleaned-up.
> 
>
> Key: HIVE-19219
> URL: https://issues.apache.org/jira/browse/HIVE-19219
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19219.01.patch
>
>
> This is the case where the events were deleted on source because of old event 
> purging and hence min(source event id) > target event id (last replicated 
> event id).
> Repl dump should fail in this case so that user can drop the database and 
> bootstrap again.
> Cleaner thread is concurrently removing the expired events from 
> NOTIFICATION_LOG table. So, it is necessary to check if the current dump 
> missed any event while dumping. It is guaranteed that cleaner would clean 
> sequentially based on event id/eventTime. So, it is enough to check if the 
> last replicated event is still there in NOTIFICATION_LOG table even after 
> dumping all events.
> If it is not found, then likely some events missed in the dump and hence 
> throw error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17477) PidFilePatternConverter not rolling log according to their pid

[Bohdan Chupika (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bohdan Chupika updated HIVE-17477:
--
Attachment: HIVE-17477.3.patch

> PidFilePatternConverter not rolling log according to their pid 
> ---
>
> Key: HIVE-17477
> URL: https://issues.apache.org/jira/browse/HIVE-17477
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Reporter: Vlad Gudikov
>Assignee: Bohdan Chupika
>Priority: Major
> Attachments: HIVE-17477.1.patch, HIVE-17477.2.patch, 
> HIVE-17477.3.patch
>
>
> We can use pid  in th filePattern:
> appender.DRFA.filePattern = 
> ${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}.%pid
> But when its time roll the logs at the end of the RollingAppender just 
> renames hive.log by applying the pattern described above and keeps logs that 
> are not related to the process this log was named by. The issue is that all 
> processes are writing to the same log and we cannot separate them by pid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19222) TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: GC overhead limit exceeded"

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441156#comment-16441156
 ] 

Aihua Xu commented on HIVE-19222:
-

[~ashutoshc], [~sershe], [~gopalv] Increasing the memory can workaround the 
issue and we can focus on the real failures. Do you think we should commit this?

> TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: 
> GC overhead limit exceeded"
> ---
>
> Key: HIVE-19222
> URL: https://issues.apache.org/jira/browse/HIVE-19222
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19222.1.patch
>
>
> TestNegativeCliDriver tests are failing with OOM recently. Not sure why. I 
> will try to increase the memory to test out.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-19163) TestAcidOnTez testGetSplitsLocks failure

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-19163.
-
   Resolution: Fixed
Fix Version/s: (was: 3.0.0)
   3.1.0

Fixed via HIVE-17645

> TestAcidOnTez testGetSplitsLocks failure
> 
>
> Key: HIVE-19163
> URL: https://issues.apache.org/jira/browse/HIVE-19163
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Reporter: Vineet Garg
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.1.0
>
>
> Error Stack:
> {code}
> java.lang.AssertionError
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidWriteIds(DbTxnManager.java:820)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyMaterializedViewRewriting(CalcitePlanner.java:2073)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1701)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1593)
>   at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
>   at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
>   at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
>   at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1398)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1414)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:434)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12056)
>   at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:314)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:287)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:645)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1655)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1602)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:252)
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:201)
>   at 
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:984)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:930)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:917)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:984)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:930)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2470)
>   at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:215)
>   at 
> org.apache.hadoop.hive.ql.TestAcidOnTez.runStatementOnDriver(TestAcidOnTez.java:879)
>   at 
> org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks(TestAcidOnTez.java:737)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at 

[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"

[Bohdan Chupika (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441148#comment-16441148
 ] 

Bohdan Chupika commented on HIVE-18902:
---

[~belugabehr] Thank you, for your reply! I see your point, fixed it in the new 
patch HIVE-18902.02.patch

> Lower Logging Level for Cleaning Up "local RawStore"
> 
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
>  Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>  
> {code:java}
> private static void cleanupRawStore() {
>   try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
>   HMSHandler.logInfo("Cleaning up thread local RawStore...");
>   rs.shutdown();
> }
>   } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
>   handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
>   }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832  INFO  
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: 
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834  INFO  
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done 
> cleaning up thread local RawStore
> {code}
> Not very helpful logging.  Please change logging levels to _debug_ or even 
> _trace_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18845) SHOW COMAPCTIONS should show host name

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18845:
--
   Resolution: Fixed
Fix Version/s: 3.1.0
 Release Note: n/a
   Status: Resolved  (was: Patch Available)

committed to master
thanks Igor for the contribution

> SHOW COMAPCTIONS should show host name
> --
>
> Key: HIVE-18845
> URL: https://issues.apache.org/jira/browse/HIVE-18845
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HIVE-18845.01.patch, HIVE-18845.02.patch
>
>
> once the job starts, the WorkerId includes the hostname submitting the job
> but before that there is no way to tell which of the Metastores in HA set up 
> has picked up a given item to compact.  Should make it obvious to know which 
> log to look at.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Yongzhi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441151#comment-16441151
 ] 

Yongzhi Chen commented on HIVE-18986:
-

[~aihuaxu], I add comments on the review board.

> Table rename will run java.lang.StackOverflowError in dataNucleus if the 
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, 
> HIVE-18986.3.patch
>
>
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2' && ... .
>  
> 2018-03-13 17:19:52,770 INFO 
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: 
> ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: 
> db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 
> 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: 
> [pool-5-thread-200]: java.lang.StackOverflowError at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-19219:

Attachment: HIVE-19219.02.patch

> Incremental REPL DUMP should throw error if requested events are cleaned-up.
> 
>
> Key: HIVE-19219
> URL: https://issues.apache.org/jira/browse/HIVE-19219
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19219.01.patch, HIVE-19219.02.patch
>
>
> This is the case where the events were deleted on source because of old event 
> purging and hence min(source event id) > target event id (last replicated 
> event id).
> Repl dump should fail in this case so that user can drop the database and 
> bootstrap again.
> Cleaner thread is concurrently removing the expired events from 
> NOTIFICATION_LOG table. So, it is necessary to check if the current dump 
> missed any event while dumping. It is guaranteed that cleaner would clean 
> sequentially based on event id/eventTime. So, it is enough to check if the 
> last replicated event is still there in NOTIFICATION_LOG table even after 
> dumping all events.
> If it is not found, then likely some events missed in the dump and hence 
> throw error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19164) TestMetastoreVersion failures

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441120#comment-16441120
 ] 

Hive QA commented on HIVE-19164:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919261/HIVE-19164.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 13445 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-19164) TestMetastoreVersion failures

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441137#comment-16441137
 ] 

Ashutosh Chauhan commented on HIVE-19164:
-

+1

> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Vineet Garg
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.02.patch, HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws 
> an exception {{No matching version found}}. 
> To fetch schema version this method executes {code:sql} SELECT FROM 
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever 
> reason execution returns empty result set resulting in the exception. Both 
> test failures are due to this exception. I tried debugging the query 
> execution but didn't really go nowhere with it. I suspect this could be due 
> to recent metastore changes. I tried reproducing this outside test but with 
> no success.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"

[Bohdan Chupika (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bohdan Chupika updated HIVE-18902:
--
Attachment: HIVE-18902.02.patch

> Lower Logging Level for Cleaning Up "local RawStore"
> 
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
>  Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>  
> {code:java}
> private static void cleanupRawStore() {
>   try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
>   HMSHandler.logInfo("Cleaning up thread local RawStore...");
>   rs.shutdown();
> }
>   } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
>   handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
>   }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832  INFO  
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: 
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834  INFO  
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done 
> cleaning up thread local RawStore
> {code}
> Not very helpful logging.  Please change logging levels to _debug_ or even 
> _trace_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19164) TestMetastoreVersion failures

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-19164:

   Resolution: Fixed
Fix Version/s: (was: 3.0.0)
   3.1.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, Vihang!

> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Vineet Garg
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19164.02.patch, HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws 
> an exception {{No matching version found}}. 
> To fetch schema version this method executes {code:sql} SELECT FROM 
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever 
> reason execution returns empty result set resulting in the exception. Both 
> test failures are due to this exception. I tried debugging the query 
> execution but didn't really go nowhere with it. I suspect this could be due 
> to recent metastore changes. I tried reproducing this outside test but with 
> no success.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19001) ALTER TABLE ADD CONSTRAINT support for CHECK constraint

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441916#comment-16441916
 ] 

Hive QA commented on HIVE-19001:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
48s{color} | {color:red} ql: The patch generated 12 new + 428 unchanged - 0 
fixed = 440 total (was 428) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10286/dev-support/hive-personality.sh
 |
| git revision | master / 878d6ee |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10286/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10286/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10286/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ALTER TABLE ADD CONSTRAINT support for CHECK constraint
> ---
>
> Key: HIVE-19001
> URL: https://issues.apache.org/jira/browse/HIVE-19001
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19001.1.patch, HIVE-19001.2.patch, 
> HIVE-19001.3.patch, HIVE-19001.4.patch
>
>
> ALTER TABLE ADD CONSTRAINT should be able to add CHECK constraint (table 
> level)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19186) Multi Table INSERT statements query has a flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440875#comment-16440875
 ] 

Hive QA commented on HIVE-19186:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 11s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10271/dev-support/hive-personality.sh
 |
| git revision | master / 5404635 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10271/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10271/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Multi Table INSERT statements query has a flaw for partitioned table when 
> INSERT INTO and INSERT OVERWRITE are used
> ---
>
> Key: HIVE-19186
> URL: https://issues.apache.org/jira/browse/HIVE-19186
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19186.01.patch, HIVE-19186.02.patch
>
>
> One problem test case is: 
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from 
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from 
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from 
> src where key >= 100 order by key asc limit 2;
> create table multi_partitioned (key int, key2 int) partitioned by (p int);
> from intermediate
> insert into table multi_partitioned partition(p=2) select p, key
> insert overwrite table multi_partitioned partition(p=1) select key, p;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-19219:

Description: 
This is the case where the events were deleted on source because of old event 
purging and hence min(source event id) > target event id (last replicated event 
id).

Repl dump should fail in this case so that user can drop the database and 
bootstrap again.

Cleaner thread is concurrently removing the expired events from 
NOTIFICATION_LOG table. So, it is necessary to check if the current dump missed 
any event while dumping. Also, it is guaranteed that cleaner would clean 
sequentially based on event id. So, it is enough to check if the first event > 
last event id is still there in NOTIFICATION_LOG table even after dumping all 
events.

If it is not found, then likely some events missed in the dump and hence throw 
error.

  was:
This is the case where the events were deleted on source because of old event 
purging and hence min(source event id) > target event id (last replicated event 
id).

Repl dump should fail in this case so that user can drop the database and 
bootstrap again.

The next incremental repl dump could check if the events fetched from source 
notification_log table is retrieved in continuous sequence with no events 
missing. If any event is missing, it should throw error.


> Incremental REPL DUMP should throw error if requested events are cleaned-up.
> 
>
> Key: HIVE-19219
> URL: https://issues.apache.org/jira/browse/HIVE-19219
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Fix For: 3.1.0
>
> Attachments: HIVE-19219.01.patch
>
>
> This is the case where the events were deleted on source because of old event 
> purging and hence min(source event id) > target event id (last replicated 
> event id).
> Repl dump should fail in this case so that user can drop the database and 
> bootstrap again.
> Cleaner thread is concurrently removing the expired events from 
> NOTIFICATION_LOG table. So, it is necessary to check if the current dump 
> missed any event while dumping. Also, it is guaranteed that cleaner would 
> clean sequentially based on event id. So, it is enough to check if the first 
> event > last event id is still there in NOTIFICATION_LOG table even after 
> dumping all events.
> If it is not found, then likely some events missed in the dump and hence 
> throw error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-17193:
--
Attachment: HIVE-17193.4.patch

> HoS: don't combine map works that are targets of different DPPs
> ---
>
> Key: HIVE-17193
> URL: https://issues.apache.org/jira/browse/HIVE-17193
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17193.1.patch, HIVE-17193.2.patch, 
> HIVE-17193.3.patch, HIVE-17193.4.patch
>
>
> Suppose {{srcpart}} is partitioned by {{ds}}. The following query can trigger 
> the issue:
> {code}
> explain
> select * from
>   (select srcpart.ds,srcpart.key from srcpart join src on srcpart.ds=src.key) 
> a
> join
>   (select srcpart.ds,srcpart.key from srcpart join src on 
> srcpart.ds=src.value) b
> on a.key=b.key;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19046) Refactor the common parts of the HiveMetastore add_partition_core and add_partitions_pspec_core methods

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441373#comment-16441373
 ] 

Sahil Takiar commented on HIVE-19046:
-

+1 LGTM pending Hive QA

> Refactor the common parts of the HiveMetastore add_partition_core and 
> add_partitions_pspec_core methods
> ---
>
> Key: HIVE-19046
> URL: https://issues.apache.org/jira/browse/HIVE-19046
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
> Attachments: HIVE-19046.1.patch
>
>
> This is a follow-up Jira of the 
> [HIVE-18696|https://issues.apache.org/jira/browse/HIVE-18696] 
> [review|https://reviews.apache.org/r/65716/].
> The biggest part of these methods use the same code. It would make sense to 
> move this code part to a common method.
> This code is almost the same in the two methods:
> {code}
> List partFutures = Lists.newArrayList();
> final Table table = tbl;
> for (final Partition part : parts) {
>   if (!part.getTableName().equals(tblName) || 
> !part.getDbName().equals(dbName)) {
> throw new MetaException("Partition does not belong to target 
> table "
> + dbName + "." + tblName + ": " + part);
>   }
>   boolean shouldAdd = startAddPartition(ms, part, ifNotExists);
>   if (!shouldAdd) {
> existingParts.add(part);
> LOG.info("Not adding partition " + part + " as it already 
> exists");
> continue;
>   }
>   final UserGroupInformation ugi;
>   try {
> ugi = UserGroupInformation.getCurrentUser();
>   } catch (IOException e) {
> throw new RuntimeException(e);
>   }
>   partFutures.add(threadPool.submit(new Callable() {
> @Override
> public Partition call() throws Exception {
>   ugi.doAs(new PrivilegedExceptionAction() {
> @Override
> public Object run() throws Exception {
>   try {
> boolean madeDir = createLocationForAddedPartition(table, 
> part);
> if (addedPartitions.put(new PartValEqWrapper(part), 
> madeDir) != null) {
>   // Technically, for ifNotExists case, we could insert 
> one and discard the other
>   // because the first one now "exists", but it seems 
> better to report the problem
>   // upstream as such a command doesn't make sense.
>   throw new MetaException("Duplicate partitions in the 
> list: " + part);
> }
> initializeAddedPartition(table, part, madeDir);
>   } catch (MetaException e) {
> throw new IOException(e.getMessage(), e);
>   }
>   return null;
> }
>   });
>   return part;
> }
>   }));
> }
> try {
>   for (Future partFuture : partFutures) {
> Partition part = partFuture.get();
> if (part != null) {
>   newParts.add(part);
> }
>   }
> } catch (InterruptedException | ExecutionException e) {
>   // cancel other tasks
>   for (Future partFuture : partFutures) {
> partFuture.cancel(true);
>   }
>   throw new MetaException(e.getMessage());
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17481) LLAP workload management

[Thai Bui (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441379#comment-16441379
 ] 

Thai Bui commented on HIVE-17481:
-

[~prasanth_j] From reading the code and just general digging around, these 3 
settings seem to fix the guaranteed tasks issue from my end, the smaller 
queries now take regular to just a bit slower when there's a big query running.

{noformat}
  "hive.llap.task.scheduler.num.schedulable.tasks.per.node": "6",
  "hive.llap.task.scheduler.preempt.independent": "true",
  "llap.plugin.endpoint.enabled": "true",
{noformat}

I force the number of schedulable tasks to 6 to match the numbers of executor 
on each LLAP daemon. 

> LLAP workload management
> 
>
> Key: HIVE-17481
> URL: https://issues.apache.org/jira/browse/HIVE-17481
> Project: Hive
>  Issue Type: New Feature
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: Workload management design doc.pdf
>
>
> This effort is intended to improve various aspects of cluster sharing for 
> LLAP. Some of these are applicable to non-LLAP queries and may later be 
> extended to all queries. Administrators will be able to specify and apply 
> policies for workload management ("resource plans") that apply to the entire 
> cluster, with only one resource plan being active at a time. The policies 
> will be created and modified using new Hive DDL statements. 
> The policies will cover:
> * Dividing the cluster into a set of (optionally, nested) query pools that 
> are each allocated a fraction of the cluster, a set query parallelism, 
> resource sharing policy between queries, and potentially others like 
> priority, etc.
> * Mapping the incoming queries into pools based on the query user, groups, 
> explicit configuration, etc.
> * Specifying rules that perform actions on queries based on counter values 
> (e.g. killing or moving queries).
> One would also be able to switch policies on a live cluster without (usually) 
> affecting running queries, including e.g. to change policies for daytime and 
> nighttime usage patterns, and other similar scenarios. The switches would be 
> safe and atomic; versioning may eventually be supported.
> Some implementation details:
> * WM will only be supported in HS2 (for obvious reasons).
> * All LLAP query AMs will run in "interactive" YARN queue and will be 
> fungible between Hive pools.
> * We will use the concept of "guaranteed tasks" (also known as ducks) to 
> enforce cluster allocation without a central scheduler and without 
> compromising throughput. Guaranteed tasks preempt other (speculative) tasks 
> and are distributed from HS2 to AMs, and from AMs to tasks, in accordance 
> with percentage allocations in the policy. Each "duck" corresponds to a CPU 
> resource on the cluster. The implementation will be isolated so as to allow 
> different ones later.
> * In future, we may consider improved task placement and late binding, 
> similar to the ones described in Sparrow paper, to work around potential 
> hotspots/etc. that are not avoided with the decentralized scheme.
> * Only one HS2 will initially be supported to avoid split-brain workload 
> management. We will also implement (in a tangential set of work items) 
> active-passive HS2 recovery. Eventually, we intend to switch to full 
> active-active HS2 configuration with shared WM and Tez session pool (unlike 
> the current case with 2 separate session pools). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441389#comment-16441389
 ] 

Sahil Takiar commented on HIVE-19204:
-

Could you create an RB?

Does the exception in {{setException}} get its full stack printed in the HS2 
logs; I see some inconsistencies where sometimes when we run {{setException}} 
we print the error on the log and sometimes we don't

> Detailed errors from some tasks are not displayed to the client because the 
> tasks don't set exception when they fail
> 
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result 
> will have such exception set and Driver.java will get such details and 
> display to the client. But some tasks don't set such exceptions so the client 
> won't see such details unless you check the HS2 log.
>   
> {noformat}
>   public void runSequential() {
> int exitVal = -101;
> try {
>   exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
>   if (tsk.getException() == null) {
> tsk.setException(t);
>   }
>   LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
>   result.setTaskError(tsk.getException());
> }
>   }
>  {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18915) Better client logging when a HoS session can't be opened

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441393#comment-16441393
 ] 

Sahil Takiar commented on HIVE-18915:
-

One comment, when creating the {{HiveException}} for 
{{SPARK_CREATE_CLIENT_ERROR}} you probably want to use 
{{Throwables#getRootCause}} as the root cause exception is often the most 
relevant.

It would be nice (but not necessary) to have a negative CLI driver test for 
this, we basically need a way to force a failure in the spark-submit code-path.

> Better client logging when a HoS session can't be opened
> 
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch, HIVE-18915.2.patch, 
> HIVE-18915.3.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client 
> for Spark session [id]}} when a HoS session can't be opened, would be better 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19104) When test MetaStore is started with retry the instances should be independent

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441401#comment-16441401
 ] 

Sahil Takiar commented on HIVE-19104:
-

+1

> When test MetaStore is started with retry the instances should be independent
> -
>
> Key: HIVE-19104
> URL: https://issues.apache.org/jira/browse/HIVE-19104
> Project: Hive
>  Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19104.2.patch, HIVE-19104.3.patch, 
> HIVE-19104.4.patch, HIVE-19104.5.patch, HIVE-19104.6.patch, 
> HIVE-19104.7.patch, HIVE-19104.patch
>
>
> When multiple MetaStore instances are started with 
> {{MetaStoreTestUtils.startMetaStoreWithRetry}} currently they use the same 
> JDBC url, and warehouse directory. This can cause problem in the tests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17970) MM LOAD DATA with OVERWRITE doesn't use base_n directory concept

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441381#comment-16441381
 ] 

Hive QA commented on HIVE-17970:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919279/HIVE-17970.02.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 14237 tests 
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed 
out) (batchId=247)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade] 
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status]
 (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_fs_overwrite] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_orc_part] 
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_overwrite] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
 (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats3] (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] 
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_dyn_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_map_operators]
 (batchId=93)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[bucket_mapjoin_wrong_table_metadata_2]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[joinneg] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ptf_negative_JoinWithAmbigousAlias]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subq_insert] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[update_notnull_constraint]
 (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats3] 
(batchId=116)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut 
(batchId=225)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitMultiInsertOverwriteTable
 (batchId=261)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=264)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion 
(batchId=264)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversionVectorized
 (batchId=264)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataPartitioned (batchId=264)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=264)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdateVectorized (batchId=264)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataVectorized (batchId=264)
org.apache.hive.service.TestDFSErrorHandling.testAccessDenied (batchId=238)

[jira] [Commented] (HIVE-17647) DDLTask.generateAddMmTasks(Table tbl) and other random code should not start transactions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441415#comment-16441415
 ] 

Hive QA commented on HIVE-17647:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
57s{color} | {color:red} ql: The patch generated 12 new + 1548 unchanged - 8 
fixed = 1560 total (was 1556) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10275/dev-support/hive-personality.sh
 |
| git revision | master / 3d1bf34 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10275/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10275/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10275/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> DDLTask.generateAddMmTasks(Table tbl) and other random code should not start 
> transactions
> -
>
> Key: HIVE-17647
> URL: https://issues.apache.org/jira/browse/HIVE-17647
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-17647.01.patch, HIVE-17647.02.patch, 
> HIVE-17647.patch
>
>
> This method (and other places) have 
> {noformat}
>   if (txnManager.isTxnOpen()) {
> mmWriteId = txnManager.getCurrentTxnId();
>   } else {
> mmWriteId = txnManager.openTxn(new Context(conf), conf.getUser());
> txnManager.commitTxn();
>   }
> {noformat}
> this should throw if there is no open transaction.  It should never open one.
> In general the logic seems suspect.  Looks like the intent is to move all 
> existing files into a delta_x_x/ when a plain table is converted to MM table. 
>  This seems like something that needs to be done from under an Exclusive lock 
> to prevent concurrent Insert operations writing data under table/partition 
> root.  But this is too 

[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441444#comment-16441444
 ] 

Naveen Gangam commented on HIVE-18778:
--

[~daijy] Any luck with the tests? Thanks

> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch, 
> HIVE-18778.2.patch, HIVE-18778.3.patch, HIVE-18778_TestCliDriver.patch, 
> HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain 
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid 
> privileges
>  Required privilege( Table) not available in input privileges
>  The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an 
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code. 
> However, these are never set on the instance of ExplainSemanticAnalyzer 
> itself and thus is not propagated into the HookContext in the calling Driver 
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses 
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this 
> code attempts to update the HookContext with the input/output info from ESA 
> which is never set.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly

[Bharathkrishna Guruvayoor Murali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali updated HIVE-19133:

Attachment: HIVE-19133.5.patch

> HS2 WebUI phase-wise performance metrics not showing correctly
> --
>
> Key: HIVE-19133
> URL: https://issues.apache.org/jira/browse/HIVE-19133
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Web UI
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19133.1.patch, HIVE-19133.2.patch, 
> HIVE-19133.3.patch, HIVE-19133.4.patch, HIVE-19133.5.patch, WebUI-compile 
> time query metrics.png
>
>
> The query specific WebUI metrics (go to drilldown -> performance logging) are 
> not showing up in the correct phase and are often mixed up.
> Attaching screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19212) Fix findbugs yetus pre-commit checks

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-19212:

Status: Patch Available  (was: Open)

> Fix findbugs yetus pre-commit checks
> 
>
> Key: HIVE-19212
> URL: https://issues.apache.org/jira/browse/HIVE-19212
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19212.1.patch
>
>
> Follow up from HIVE-18883, the committed patch isn't working and Findbugs is 
> still not working.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly

[Bharathkrishna Guruvayoor Murali (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441376#comment-16441376
 ] 

Bharathkrishna Guruvayoor Murali commented on HIVE-19133:
-

Noticed some tests failing as result of removing driver.run.
Fixing them.

> HS2 WebUI phase-wise performance metrics not showing correctly
> --
>
> Key: HIVE-19133
> URL: https://issues.apache.org/jira/browse/HIVE-19133
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Web UI
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19133.1.patch, HIVE-19133.2.patch, 
> HIVE-19133.3.patch, HIVE-19133.4.patch, HIVE-19133.5.patch, WebUI-compile 
> time query metrics.png
>
>
> The query specific WebUI metrics (go to drilldown -> performance logging) are 
> not showing up in the correct phase and are often mixed up.
> Attaching screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19196) TestTriggersMoveWorkloadManager is flaky

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441404#comment-16441404
 ] 

Sergey Shelukhin commented on HIVE-19196:
-

I can repro the failures very occasionally when running in a loop, however I 
can see the WM events are the same in both failed and successful log file. 
Trying to add some more logging to see what happens... test seems to be timing 
dependent.

> TestTriggersMoveWorkloadManager is flaky
> 
>
> Key: HIVE-19196
> URL: https://issues.apache.org/jira/browse/HIVE-19196
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
>Priority: Major
>
> This is a flaky test which randomly fails. Consider improving its stability.
> {code}
> org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
> Failing for the past 1 build (Since Failed#10161 )
> Took 2.4 sec.
> Error Message
> '"eventType" : "GET"' expected in STDERR capture, but not found.
> Stacktrace
> java.lang.AssertionError: '"eventType" : "GET"' expected in STDERR capture, 
> but not found.
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> org.apache.hive.jdbc.AbstractJdbcTriggersTest.runQueryWithTrigger(AbstractJdbcTriggersTest.java:169)
>   at 
> org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill(TestTriggersMoveWorkloadManager.java:261)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19126) CachedStore: Use memory estimation to limit cache size during prewarm

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-19126:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to branch-3 and master. Thanks [~thejas].

> CachedStore: Use memory estimation to limit cache size during prewarm
> -
>
> Key: HIVE-19126
> URL: https://issues.apache.org/jira/browse/HIVE-19126
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19126.1.patch, HIVE-19126.2.patch, 
> HIVE-19126.3.patch, HIVE-19126.4.patch, HIVE-19126.5.patch, 
> HIVE-19126.5.patch, HIVE-19126.6.patch
>
>
> We can rely on 
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
>  to estimate memory of SharedCache. This jira addresses the size estimation 
> during prewarm, so that we can stop when we hit the memory limit. In a 
> follow-up jira, we will work on estimation/eviction after prewarm is 
> complete, so that we can keep the frequently used tables and their partitions 
> in cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19141) TestNegativeCliDriver insert_into_notnull_constraint, insert_into_acid_notnull failing

[Igor Kryvenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igor Kryvenko updated HIVE-19141:
-
Attachment: HIVE-19141.03.patch

> TestNegativeCliDriver insert_into_notnull_constraint, 
> insert_into_acid_notnull failing
> --
>
> Key: HIVE-19141
> URL: https://issues.apache.org/jira/browse/HIVE-19141
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19141.01.patch, HIVE-19141.02.patch, 
> HIVE-19141.03.patch
>
>
> These tests have been consistently failing for a while. I suspect HIVE-18727 
> has caused these failures. HIVE-18727 changed the code to throw ERROR instead 
> of EXCEPTION if constraints are violated. I guess Negative cli driver doesn't 
> handle errors.
> Following are full list of related failures:
> TestNegativeCliDriver.alter_notnull_constraint_violation
> TestNegativeCliDriver.insert_into_acid_notnull 
> TestNegativeCliDriver.insert_into_notnull_constraint 
> TestNegativeCliDriver.insert_multi_into_notnull 
> TestNegativeCliDriver.insert_overwrite_notnull_constraint 
> TestNegativeCliDriver.update_notnull_constraint



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19222) TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: GC overhead limit exceeded"

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441197#comment-16441197
 ] 

Ashutosh Chauhan commented on HIVE-19222:
-

Yeah.. I think so. 1 GB memory should be sufficient to run -ve tests, so it 
does look like there is a leak. But having tests fail on every run is not 
useful either. So, lets get this in. We can use HIVE-19000 for leak. 

> TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: 
> GC overhead limit exceeded"
> ---
>
> Key: HIVE-19222
> URL: https://issues.apache.org/jira/browse/HIVE-19222
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19222.1.patch
>
>
> TestNegativeCliDriver tests are failing with OOM recently. Not sure why. I 
> will try to increase the memory to test out.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19222) TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: GC overhead limit exceeded"

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-19222:

   Resolution: Fixed
Fix Version/s: 3.1.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, Aihua!

> TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: 
> GC overhead limit exceeded"
> ---
>
> Key: HIVE-19222
> URL: https://issues.apache.org/jira/browse/HIVE-19222
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19222.1.patch
>
>
> TestNegativeCliDriver tests are failing with OOM recently. Not sure why. I 
> will try to increase the memory to test out.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19227) Update golden files for negative tests

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-19227:

Attachment: HIVE-19227.patch

> Update golden files for negative tests
> --
>
> Key: HIVE-19227
> URL: https://issues.apache.org/jira/browse/HIVE-19227
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-19227.patch
>
>
> Now that we are able to run TestNegativeCliDriver some of golden files are 
> found to be outdated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19227) Update golden files for negative tests

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-19227:

Status: Patch Available  (was: Open)

> Update golden files for negative tests
> --
>
> Key: HIVE-19227
> URL: https://issues.apache.org/jira/browse/HIVE-19227
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-19227.patch
>
>
> Now that we are able to run TestNegativeCliDriver some of golden files are 
> found to be outdated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from Acid table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441272#comment-16441272
 ] 

Hive QA commented on HIVE-18739:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919270/HIVE-18739.25.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 13492 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18739) Add support for Export from Acid table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441195#comment-16441195
 ] 

Hive QA commented on HIVE-18739:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  1m  
0s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} common: The patch generated 1 new + 7 unchanged - 0 
fixed = 8 total (was 7) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
54s{color} | {color:red} ql: The patch generated 83 new + 1310 unchanged - 13 
fixed = 1393 total (was 1323) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
26s{color} | {color:red} standalone-metastore: The patch generated 1 new + 679 
unchanged - 0 fixed = 680 total (was 679) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m 
14s{color} | {color:red} standalone-metastore generated 1 new + 53 unchanged - 
0 fixed = 54 total (was 53) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10273/dev-support/hive-personality.sh
 |
| git revision | master / 5404635 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus/diff-checkstyle-common.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus/diff-checkstyle-standalone-metastore.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus/whitespace-tabs.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus/diff-javadoc-javadoc-standalone-metastore.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus/patch-asflicense-problems.txt
 |
| modules | C: common ql standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10273/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene 

[jira] [Commented] (HIVE-19186) Multi Table INSERT statements query has a flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441230#comment-16441230
 ] 

Steve Yeom commented on HIVE-19186:
---

Checked 2 failed tests of age 1, but are clear in my environment with the patch 
version 2:
default_constraint.q and stats_publisher_errors_2.q.

Will continue to check.

> Multi Table INSERT statements query has a flaw for partitioned table when 
> INSERT INTO and INSERT OVERWRITE are used
> ---
>
> Key: HIVE-19186
> URL: https://issues.apache.org/jira/browse/HIVE-19186
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19186.01.patch, HIVE-19186.02.patch
>
>
> One problem test case is: 
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from 
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from 
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from 
> src where key >= 100 order by key asc limit 2;
> create table multi_partitioned (key int, key2 int) partitioned by (p int);
> from intermediate
> insert into table multi_partitioned partition(p=2) select p, key
> insert overwrite table multi_partitioned partition(p=1) select key, p;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19229) Automatically Convert VARCHAR to STRING

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-19229:
---
Labels: incompatibleChange  (was: )

> Automatically Convert VARCHAR to STRING
> ---
>
> Key: HIVE-19229
> URL: https://issues.apache.org/jira/browse/HIVE-19229
> Project: Hive
>  Issue Type: New Feature
>  Components: Database/Schema, HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Priority: Major
>  Labels: incompatibleChange
>
> Using {{STRING}} datatypes is much faster than {{VARCHAR}}.  Add a 
> configuration to HiveServer2 (HS2) that automatically converts {{VARCHAR}} 
> DDL table definitions to {{STRING}}.  In this way, if a BI tool is creating 
> {{VARCHAR}} column in tables, and the behavior cannot be changed, HS2 will 
> change them automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17970) MM LOAD DATA with OVERWRITE doesn't use base_n directory concept

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441317#comment-16441317
 ] 

Hive QA commented on HIVE-17970:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
44s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
18s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 15 
unchanged - 1 fixed = 16 total (was 16) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
46s{color} | {color:red} ql: The patch generated 4 new + 403 unchanged - 13 
fixed = 407 total (was 416) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 53s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10274/dev-support/hive-personality.sh
 |
| git revision | master / 9380dcb |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10274/yetus/patch-mvninstall-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10274/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10274/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10274/yetus/whitespace-eol.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10274/yetus/patch-asflicense-problems.txt
 |
| modules | C: common itests itests/hive-unit ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10274/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> MM LOAD DATA with OVERWRITE doesn't use base_n directory concept
> 
>
> Key: HIVE-17970
> URL: https://issues.apache.org/jira/browse/HIVE-17970
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-17970.01.patch, HIVE-17970.02.patch, 
> HIVE-17970.patch
>
>
> Judging by 
> {code:java}
> Hive.loadTable(Path loadPath, String tableName, LoadFileType 

[jira] [Commented] (HIVE-19048) Initscript errors are ignored

[Bharathkrishna Guruvayoor Murali (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441329#comment-16441329
 ] 

Bharathkrishna Guruvayoor Murali commented on HIVE-19048:
-

Hi [~vihangk1], can you kindly take a final look at this patch and push it if 
it looks fine.

> Initscript errors are ignored
> -
>
> Key: HIVE-19048
> URL: https://issues.apache.org/jira/browse/HIVE-19048
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Reporter: Zoltan Haindrich
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19048.1.patch, HIVE-19048.2.patch
>
>
> I've been running some queries for a while when I've noticed that my 
> initscript has an error; and beeline stops interpreting the initscript after 
> encountering the first error.
> {code}
> echo 'invalid;' > init.sql
> echo 'select 1;' > s1.sql
> beeline -u jdbc:hive2://localhost:1/ -n hive -i init.sql -f s1.sql 
> [...]
> Running init script init.sql
> 0: jdbc:hive2://localhost:1/> invalid;
> Error: Error while compiling statement: FAILED: ParseException line 1:0 
> cannot recognize input near 'invalid' '' '' (state=42000,code=4)
> 0: jdbc:hive2://localhost:1/> select 1;
> [...]
> $ echo $?
> 0
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19227) Update golden files for negative tests

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan reassigned HIVE-19227:
---


> Update golden files for negative tests
> --
>
> Key: HIVE-19227
> URL: https://issues.apache.org/jira/browse/HIVE-19227
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Now that we are able to run TestNegativeCliDriver some of golden files are 
> found to be outdated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19228) Remove commons-httpclient 3.x usage

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani reassigned HIVE-19228:
--


> Remove commons-httpclient 3.x usage
> ---
>
> Key: HIVE-19228
> URL: https://issues.apache.org/jira/browse/HIVE-19228
> Project: Hive
>  Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
>
> Commons-httpclient is not supported well anymore.  Remove dependency and move 
> to Apache HTTP client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Export from Acid table

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441325#comment-16441325
 ] 

Sergey Shelukhin commented on HIVE-18739:
-

+1

> Add support for Export from Acid table
> --
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.06.patch, HIVE-18739.08.patch, HIVE-18739.09.patch, 
> HIVE-18739.10.patch, HIVE-18739.11.patch, HIVE-18739.12.patch, 
> HIVE-18739.13.patch, HIVE-18739.14.patch, HIVE-18739.15.patch, 
> HIVE-18739.16.patch, HIVE-18739.17.patch, HIVE-18739.19.patch, 
> HIVE-18739.20.patch, HIVE-18739.21.patch, HIVE-18739.23.patch, 
> HIVE-18739.24.patch, HIVE-18739.25.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19227) Update golden files for negative tests

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-19227:

   Resolution: Fixed
Fix Version/s: 3.1.0
   Status: Resolved  (was: Patch Available)

Pushed to master.

> Update golden files for negative tests
> --
>
> Key: HIVE-19227
> URL: https://issues.apache.org/jira/browse/HIVE-19227
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19227.patch
>
>
> Now that we are able to run TestNegativeCliDriver some of golden files are 
> found to be outdated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19000) Fix TestNegativeCliDriver

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441202#comment-16441202
 ] 

Ashutosh Chauhan commented on HIVE-19000:
-

HIVE-19222 worked around failures by increasing memory for tests. But doesn't 
fix leak. Lets use this jira for leak investigation.

> Fix TestNegativeCliDriver
> -
>
> Key: HIVE-19000
> URL: https://issues.apache.org/jira/browse/HIVE-19000
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Thejas M Nair
>Priority: Major
>
> TestNegativeCliDriver is failing since a while. We should investigate and fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-12971) Hive Support for Kudu

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani reassigned HIVE-12971:
--

Assignee: Janaki Lahorani  (was: bimal tandel)

> Hive Support for Kudu
> -
>
> Key: HIVE-12971
> URL: https://issues.apache.org/jira/browse/HIVE-12971
> Project: Hive
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Lenni Kuff
>Assignee: Janaki Lahorani
>Priority: Major
>
> JIRA for tracking work related to Hive/Kudu integration.
> It would be useful to allow Kudu data to be accessible via Hive. This would 
> involve creating a Kudu SerDe/StorageHandler and implementing support for 
> QUERY and DML commands like SELECT, INSERT, UPDATE, and DELETE. Kudu 
> Input/OutputFormats classes already exist. The work can be staged to support 
> this functionality incrementally.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18915) Better client logging when a HoS session can't be opened

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441593#comment-16441593
 ] 

Aihua Xu commented on HIVE-18915:
-

[~stakiar] Do you know a way to force a failure? I used a unit test to cover 
that. 



> Better client logging when a HoS session can't be opened
> 
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch, HIVE-18915.2.patch, 
> HIVE-18915.3.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client 
> for Spark session [id]}} when a HoS session can't be opened, would be better 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18915) Better client logging when a HoS session can't be opened

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-18915:

Status: In Progress  (was: Patch Available)

> Better client logging when a HoS session can't be opened
> 
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch, HIVE-18915.2.patch, 
> HIVE-18915.3.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client 
> for Spark session [id]}} when a HoS session can't be opened, would be better 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19234) Fix flaky tests (see description)

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal reassigned HIVE-19234:
-


> Fix flaky tests (see description)
> -
>
> Key: HIVE-19234
> URL: https://issues.apache.org/jira/browse/HIVE-19234
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
>
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
>  is still failing with plan differences. Age : 161
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] fails with 
> exception. Going by the name it should not even run on TestCliDriver.
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] : fails 
> with exception. Going by the name it should not even run on TestCliDriver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19195) Fix flaky tests and cleanup testconfiguration to run llap specific tests in llap only.

[Deepak Jaiswal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441599#comment-16441599
 ] 

Deepak Jaiswal commented on HIVE-19195:
---

[~ashutoshc] thanks for the analysis. The last two should be easy to fix. I 
will take a look why tez_smb_1 is giving different plan again.

 

I have opened HIVE-19234 to track this.

> Fix flaky tests and cleanup testconfiguration to run llap specific tests in 
> llap only.
> --
>
> Key: HIVE-19195
> URL: https://issues.apache.org/jira/browse/HIVE-19195
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Deepak Jaiswal
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19195.1.patch
>
>
> This test is certainly flaky. Seems like it makes some assumption about 
> available memory. Consider dropping this altogether.
>  Also remove tests from llaplocal.shared list to llaplocal so that they dont 
> run in MR.
> Makes HIVE-17055 redundant.
> {code:java}
> Client Execution succeeded but contained differences (error code = 1) after 
> executing auto_sortmerge_join_2.q 
> 1101,1103d1100
> < Hive Runtime Error: Map local work exhausted memory
> < FAILED: Execution Error, return code 3 from 
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
> < ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441605#comment-16441605
 ] 

Aihua Xu commented on HIVE-19204:
-

setException doesn't print the log to log file but it's populated the exception 
to the client. LOG.error() is the way to print to the log file. 

> Detailed errors from some tasks are not displayed to the client because the 
> tasks don't set exception when they fail
> 
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result 
> will have such exception set and Driver.java will get such details and 
> display to the client. But some tasks don't set such exceptions so the client 
> won't see such details unless you check the HS2 log.
>   
> {noformat}
>   public void runSequential() {
> int exitVal = -101;
> try {
>   exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
>   if (tsk.getException() == null) {
> tsk.setException(t);
>   }
>   LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
>   result.setTaskError(tsk.getException());
> }
>   }
>  {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-19204:

Attachment: (was: HIVE-19204.1.patch)

> Detailed errors from some tasks are not displayed to the client because the 
> tasks don't set exception when they fail
> 
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result 
> will have such exception set and Driver.java will get such details and 
> display to the client. But some tasks don't set such exceptions so the client 
> won't see such details unless you check the HS2 log.
>   
> {noformat}
>   public void runSequential() {
> int exitVal = -101;
> try {
>   exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
>   if (tsk.getException() == null) {
> tsk.setException(t);
>   }
>   LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
>   result.setTaskError(tsk.getException());
> }
>   }
>  {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19131) DecimalColumnStatsMergerTest comparison review

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-19131:

Status: Patch Available  (was: Open)

> DecimalColumnStatsMergerTest comparison review
> --
>
> Key: HIVE-19131
> URL: https://issues.apache.org/jira/browse/HIVE-19131
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-19131.01.patch
>
>
> DecimalColumnStatsMergerTest has a strange comparison logic, which needs to 
> be reviewed.
> Regarding low and high values, it uses compareTo with the same direction, 
> which seems to be incorrect: old.compareTo(new) > 0 -> pick old value in both 
> cases
> {code:java}
> Decimal lowValue = aggregateData.getLowValue() != null && 
> (aggregateData.getLowValue().compareTo(newData.getLowValue()) > 0) ? 
> aggregateData .getLowValue() : newData.getLowValue(); 
> aggregateData.setLowValue(lowValue); 
> Decimal highValue = aggregateData.getHighValue() != null && 
> (aggregateData.getHighValue().compareTo(newData.getHighValue()) > 0) ? 
> aggregateData .getHighValue() : newData.getHighValue();
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19131) DecimalColumnStatsMergerTest comparison review

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-19131:

Attachment: HIVE-19131.01.patch

> DecimalColumnStatsMergerTest comparison review
> --
>
> Key: HIVE-19131
> URL: https://issues.apache.org/jira/browse/HIVE-19131
> Project: Hive
>  Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-19131.01.patch
>
>
> DecimalColumnStatsMergerTest has a strange comparison logic, which needs to 
> be reviewed.
> Regarding low and high values, it uses compareTo with the same direction, 
> which seems to be incorrect: old.compareTo(new) > 0 -> pick old value in both 
> cases
> {code:java}
> Decimal lowValue = aggregateData.getLowValue() != null && 
> (aggregateData.getLowValue().compareTo(newData.getLowValue()) > 0) ? 
> aggregateData .getLowValue() : newData.getLowValue(); 
> aggregateData.setLowValue(lowValue); 
> Decimal highValue = aggregateData.getHighValue() != null && 
> (aggregateData.getHighValue().compareTo(newData.getHighValue()) > 0) ? 
> aggregateData .getHighValue() : newData.getHighValue();
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HIVE-19195) Fix flaky tests and cleanup testconfiguration to run llap specific tests in llap only.

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan reopened HIVE-19195:
-

 org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] 
is still failing with plan differences. Age : 161

org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] fails with 
exception. Going by the name it should not even run on TestCliDriver.
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] : fails 
with exception. Going by the name it should not even run on TestCliDriver.

> Fix flaky tests and cleanup testconfiguration to run llap specific tests in 
> llap only.
> --
>
> Key: HIVE-19195
> URL: https://issues.apache.org/jira/browse/HIVE-19195
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Deepak Jaiswal
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19195.1.patch
>
>
> This test is certainly flaky. Seems like it makes some assumption about 
> available memory. Consider dropping this altogether.
>  Also remove tests from llaplocal.shared list to llaplocal so that they dont 
> run in MR.
> Makes HIVE-17055 redundant.
> {code:java}
> Client Execution succeeded but contained differences (error code = 1) after 
> executing auto_sortmerge_join_2.q 
> 1101,1103d1100
> < Hive Runtime Error: Map local work exhausted memory
> < FAILED: Execution Error, return code 3 from 
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
> < ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-19233:
-


> Add utility for acid 1.0 to 2.0 migration
> -
>
> Key: HIVE-19233
> URL: https://issues.apache.org/jira/browse/HIVE-19233
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)