ptests are broken

2017-10-19 Thread Zoltan Haindrich 

Hello,

It seems like ptests are broken down :)
From the logs I suspect that the problem is possibly caused by the 
version change of testutils/ptest2/pom.xml from 1.0 to 3.0.
Could someone who has access to the jenkins job configuration change the 
java classpath to include hive-ptest-3.0-classes.jar?


cheers,
Zoltan

[jira] [Created] (HIVE-17863) Vectorization: Two Q files produce wrong PTF query results

2017-10-19 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-17863:
---

 Summary: Vectorization: Two Q files produce wrong PTF query results
 Key: HIVE-17863
 URL: https://issues.apache.org/jira/browse/HIVE-17863
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical


vector_windowing_multipartitioning.q
vector_windowing_order_null.q



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[VOTE] Apache Hive 2.3.1 Release Candidate 0

2017-10-19 Thread Jesus Camacho Rodriguez 
Apache Hive 2.3.1 Release Candidate 0 is available here:
http://people.apache.org/~jcamacho/hive-2.3.1-rc0/

Maven artifacts are available here:
https://repository.apache.org/content/repositories/orgapachehive-1081/

Source tag for RC0 is at:
https://github.com/apache/hive/releases/tag/release-2.3.1-rc0/

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

[jira] [Created] (HIVE-17862) Update copyright date in NOTICE

2017-10-19 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-17862:
--

 Summary: Update copyright date in NOTICE
 Key: HIVE-17862
 URL: https://issues.apache.org/jira/browse/HIVE-17862
 Project: Hive
  Issue Type: Task
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Priority: Trivial
 Fix For: 2.3.1






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17861) MM tables - multi-IOW may be broken

2017-10-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17861:
---

 Summary: MM tables - multi-IOW may be broken
 Key: HIVE-17861
 URL: https://issues.apache.org/jira/browse/HIVE-17861
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


After HIVE-17856, see if multi IOW was commented out because of IOW issues or 
because it's broken in addition to IOW being broken.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17860) verify CTAS logic for MM tables

2017-10-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17860:
---

 Summary: verify CTAS logic for MM tables
 Key: HIVE-17860
 URL: https://issues.apache.org/jira/browse/HIVE-17860
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


The mm_all test output matches the output before integration, but we need to 
check the internals wrt txns make sense (and perhaps improve the test)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17859) MM tables - Tez merge may not run

2017-10-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17859:
---

 Summary: MM tables - Tez merge may not run
 Key: HIVE-17859
 URL: https://issues.apache.org/jira/browse/HIVE-17859
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


If mm_all test is executed on MiniLlap, all the merge test cases pass but with 
changed order of rows; however, one can see in stats_mm, etc cases that while 
the stats are correct, the number of files changes. Seems like merge doesn't 
work in this case where it worked before. It's not pertinent to stats cases but 
it should be examined. Perhaps stats output for the # of files should be added 
to merge cases to make sure merge actually merges.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17858) MM - some union cases are broken

2017-10-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17858:
---

 Summary: MM - some union cases are broken
 Key: HIVE-17858
 URL: https://issues.apache.org/jira/browse/HIVE-17858
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


mm_all test no longer runs on LLAP; if it's executed in LLAP, one can see that 
some union cases no longer work.
Queries on partunion_mm, skew_dp_union_mm produce no results.

I'm not sure what part of "integration" broke it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17857) Upgrade to orc 1.4

2017-10-19 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-17857:
---

 Summary: Upgrade to orc 1.4
 Key: HIVE-17857
 URL: https://issues.apache.org/jira/browse/HIVE-17857
 Project: Hive
  Issue Type: Task
  Components: ORC
Reporter: Ashutosh Chauhan






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17856) MM tables - IOW is broken

2017-10-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17856:
---

 Summary: MM tables - IOW is broken
 Key: HIVE-17856
 URL: https://issues.apache.org/jira/browse/HIVE-17856
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


The following tests were removed from mm_all 
{noformat}
drop table iow0_mm;
create table iow0_mm(key int) tblproperties("transactional"="true", 
"transactional_properties"="insert_only");
insert overwrite table iow0_mm select key from intermediate;
insert into table iow0_mm select key + 1 from intermediate;
select * from iow0_mm order by key;
insert overwrite table iow0_mm select key + 2 from intermediate;
select * from iow0_mm order by key;
drop table iow0_mm;


drop table iow1_mm; 
create table iow1_mm(key int) partitioned by (key2 int)  
tblproperties("transactional"="true", "transactional_properties"="insert_only");
insert overwrite table iow1_mm partition (key2)
select key as k1, key from intermediate union all select key as k1, key from 
intermediate;
insert into table iow1_mm partition (key2)
select key + 1 as k1, key from intermediate union all select key as k1, key 
from intermediate;
select * from iow1_mm order by key, key2;
insert overwrite table iow1_mm partition (key2)
select key + 3 as k1, key from intermediate union all select key + 4 as k1, key 
from intermediate;
select * from iow1_mm order by key, key2;
insert overwrite table iow1_mm partition (key2)
select key + 3 as k1, key + 3 from intermediate union all select key + 2 as k1, 
key + 2 from intermediate;
select * from iow1_mm order by key, key2;
drop table iow1_mm;
{noformat}

{noformat}
drop table simple_mm;
create table simple_mm(key int) stored as orc tblproperties 
("transactional"="true", "transactional_properties"="insert_only");
insert into table simple_mm select key from intermediate;
-insert overwrite table simple_mm select key from intermediate;
{noformat}





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17855) conversion to MM tables via alter may be broken

2017-10-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17855:
---

 Summary: conversion to MM tables via alter may be broken
 Key: HIVE-17855
 URL: https://issues.apache.org/jira/browse/HIVE-17855
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


{noformat}
git difftool 77511070dd 77511070dd^ -- */mm_conversions.q
{noformat}
Looks like during ACID "integration" alter was simply quietly changed to 
create+insert, because it's broken.
I asked to keep feature parity with every change but I should have rather 
insisted on it and -1d all the patches that didn't... This is just annoying. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17854) LlapRecordReader should have getRowNumber() like org.apache.orc.RecordReader

2017-10-19 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-17854:
-

 Summary: LlapRecordReader should have getRowNumber() like 
org.apache.orc.RecordReader
 Key: HIVE-17854
 URL: https://issues.apache.org/jira/browse/HIVE-17854
 Project: Hive
  Issue Type: Bug
  Components: llap, Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman


getRowNumber() is required to read data in non-acid tables that were converted 
to acid.

Currently when VectorizedOrcAcidRowBatchReader is used from LlapRecordReader 
this functionality is not available making it impossible to vectorize 
reads/updates of non-aicd-to-acid tables with LLAP (until major compaction) in 
the presence of any deletes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17853) RetryingMetaStoreClient loses UGI impersonation-context when reconnecting after timeout

2017-10-19 Thread Mithun Radhakrishnan (JIRA) 
Mithun Radhakrishnan created HIVE-17853:
---

 Summary: RetryingMetaStoreClient loses UGI impersonation-context 
when reconnecting after timeout
 Key: HIVE-17853
 URL: https://issues.apache.org/jira/browse/HIVE-17853
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0, 2.4.0, 2.2.1
Reporter: Mithun Radhakrishnan
Assignee: Chris Drome
Priority: Critical


The {{RetryingMetaStoreClient}} is used to automatically reconnect to the Hive 
metastore, after client timeout, transparently to the user.

In case of user impersonation (e.g. Oozie super-user {{oozie}} impersonating a 
Hadoop user {{mithun}}, to run a workflow), in case of timeout, we find that 
the reconnect causes the {{UGI.doAs()}} context to be lost. Any further 
metastore operations will be attempted as the login-user ({{oozie}}), as 
opposed to the effective user ({{mithunr}}).

We should have a fix for this shortly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0

2017-10-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17852:
---

 Summary: remove support for list bucketing "stored as directories" 
in 3.0
 Key: HIVE-17852
 URL: https://issues.apache.org/jira/browse/HIVE-17852
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


>From the email thread:

1) LB, when stored as directories, adds a lot of low-level complexity to Hive 
tables that has to be accounted for in many places in the code where the files 
are written or modified - from FSOP to ACID/replication/export.
2) While working on some FSOP code I noticed that some of that logic is broken 
- e.g. the duplicate file removal from tasks, a pretty fundamental correctness 
feature in Hive, may be broken. LB also doesn’t appear to be compatible with 
e.g. regular bucketing.
3) The feature hasn’t seen development activity in a while; it also doesn’t 
appear to be used a lot.

Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing we 
remove it.

(2) also suggested that, if needed, it might be easier to implement similar 
functionality by adding some flexibility to partitions (which LB directories 
look like anyway); that would also keep the logic on a higher level of 
abstraction (split generation, partition pruning) as opposed to many low-level 
places like FSOP, etc. 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17851) Bucket Map Join : Pick correct number of buckets

2017-10-19 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-17851:
-

 Summary: Bucket Map Join : Pick correct number of buckets
 Key: HIVE-17851
 URL: https://issues.apache.org/jira/browse/HIVE-17851
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


CREATE TABLE tab_part (key int, value string) PARTITIONED BY(ds STRING) 
CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
CREATE TABLE tab(key int, value string) PARTITIONED BY(ds STRING) CLUSTERED BY 
(key) INTO 4 BUCKETS STORED AS TEXTFILE;

select a.key, a.value, b.value
from tab a join tab_part b on a.key = b.key;

In above case, if tab_part is bigger then it should be the streaming side and 
the smaller side should create two hash tables. However, currently, it blindly 
picks 4 as number of buckets as it is the maximum number of buckets among all 
the tables involved in the join and create 4 hash tables.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17850) can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done?

2017-10-19 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-17850:
-

 Summary: can VectorizedOrcAcidRowReader be removed once HIVE-17458 
is done?
 Key: HIVE-17850
 URL: https://issues.apache.org/jira/browse/HIVE-17850
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Priority: Minor


I don't think it will ever even be created once HIVE-17458 is done



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 63161: HIVE-17849: alterPartition() may fail to rollback transaction

2017-10-19 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63161/
---

Review request for hive, Aihua Xu, Andrew Sherman, Sahil Takiar, and Vihang 
Karajgaonkar.


Bugs: HIVE-17849
https://issues.apache.org/jira/browse/HIVE-17849


Repository: hive-git


Description
---

HIVE-17849: alterPartition() may fail to rollback transaction


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
15f2597daf79a00ad4904790eeea42582efe8646 


Diff: https://reviews.apache.org/r/63161/diff/1/


Testing
---


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-17849) alterPartition() may fail to rollback transaction

2017-10-19 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-17849:
-

 Summary: alterPartition() may fail to rollback transaction
 Key: HIVE-17849
 URL: https://issues.apache.org/jira/browse/HIVE-17849
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.3.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


In HiveAlterHandle.alterPartition() there is this code:

{code}
  try {
msdb.openTransaction();
msdb.alterPartition(dbname, name, new_part.getValues(), oldPart);
if (transactionalListeners != null && 
!transactionalListeners.isEmpty()) {
  MetaStoreListenerNotifier.notifyEvent(transactionalListeners,

EventMessage.EventType.ALTER_PARTITION,
new 
AlterPartitionEvent(new_part, oldPart, tbl, success, handler));
}

revertMetaDataTransaction = msdb.commitTransaction();
  } catch (Exception ex2) {
LOG.error("Attempt to revert partition metadata change failed. The 
revert was attempted " +
"because associated filesystem rename operation failed with 
exception " + ex.getMessage(), ex2);
if (!revertMetaDataTransaction) {
  msdb.rollbackTransaction();
}
  }
{code}

Note that there is no {{finally}} clause, so it is possible for some unchecked 
exception to occur in which case the transaction will remain active. Once this 
happens, all subsequent transactions on this thread will not behave correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17848) Bucket Map Join : Implement an efficient way to minimize loading hash table

2017-10-19 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-17848:
-

 Summary: Bucket Map Join : Implement an efficient way to minimize 
loading hash table
 Key: HIVE-17848
 URL: https://issues.apache.org/jira/browse/HIVE-17848
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


In bucket mapjoin, each task loads its own copy of hash table which is 
inefficient as load is IO heavy and due to multiple copies of same hash table, 
the tables may get GCed on a busy system.
Implement a subcache with softreference to each hash table corresponding to its 
bucketID such that it can be reused by a task.

This needs changes from Tez side to push bucket id to TezProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63133: HIVE-17054

2017-10-19 Thread Jesús Camacho Rodríguez 


> On Oct. 19, 2017, 5:59 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/groupby_join_pushdown.q.out
> > Lines 1517 (patched)
> > 
> >
> > sorry misread the plan.

No worries!


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63133/#review188718
---


On Oct. 19, 2017, 12:52 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63133/
> ---
> 
> (Updated Oct. 19, 2017, 12:52 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17054
> https://issues.apache.org/jira/browse/HIVE-17054
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17054
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 044bba1f71199cbbbd996c7e88e1eed1c0a15c21 
>   ql/src/test/queries/clientpositive/groupby_join_pushdown.q 
> a6e25685561e022db8b35e484801d874b22a03ce 
>   ql/src/test/results/clientpositive/groupby_join_pushdown.q.out 
> 95d499afef65271edbd3418b67f95691639208a2 
> 
> 
> Diff: https://reviews.apache.org/r/63133/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63133: HIVE-17054

2017-10-19 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63133/#review188718
---




ql/src/test/results/clientpositive/groupby_join_pushdown.q.out
Lines 1517 (patched)


sorry misread the plan.


- Ashutosh Chauhan


On Oct. 19, 2017, 12:52 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63133/
> ---
> 
> (Updated Oct. 19, 2017, 12:52 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17054
> https://issues.apache.org/jira/browse/HIVE-17054
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17054
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 044bba1f71199cbbbd996c7e88e1eed1c0a15c21 
>   ql/src/test/queries/clientpositive/groupby_join_pushdown.q 
> a6e25685561e022db8b35e484801d874b22a03ce 
>   ql/src/test/results/clientpositive/groupby_join_pushdown.q.out 
> 95d499afef65271edbd3418b67f95691639208a2 
> 
> 
> Diff: https://reviews.apache.org/r/63133/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63133: HIVE-17054

2017-10-19 Thread Jesús Camacho Rodríguez 


> On Oct. 19, 2017, 5:33 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/groupby_join_pushdown.q
> > Lines 69 (patched)
> > 
> >
> > Comment says in both cases optimization didn't trigger. Incorrect 
> > comment?

It is correct: it will not be triggered for PKs and for UNIQUE (L65 drops the 
PK constraint and L67 creates a UNIQUE constraint).

Btw, the plan with the GBs pushed is just above the new plans.


> On Oct. 19, 2017, 5:33 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/groupby_join_pushdown.q.out
> > Lines 1517 (patched)
> > 
> >
> > Is this expected? Either GBy needs to be in complete mode or there 
> > needs to be another reducer following this.

Stage 2 contains the Reducer to follow this, result is expected as GBy ops are 
not pushed to the inputs because column is already unique.


> On Oct. 19, 2017, 5:33 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/groupby_join_pushdown.q.out
> > Lines 1632 (patched)
> > 
> >
> > Is this expected? Either GBy needs to be in complete mode or there 
> > needs to be another reducer following this.

Same as above.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63133/#review188704
---


On Oct. 19, 2017, 12:52 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63133/
> ---
> 
> (Updated Oct. 19, 2017, 12:52 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17054
> https://issues.apache.org/jira/browse/HIVE-17054
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17054
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 044bba1f71199cbbbd996c7e88e1eed1c0a15c21 
>   ql/src/test/queries/clientpositive/groupby_join_pushdown.q 
> a6e25685561e022db8b35e484801d874b22a03ce 
>   ql/src/test/results/clientpositive/groupby_join_pushdown.q.out 
> 95d499afef65271edbd3418b67f95691639208a2 
> 
> 
> Diff: https://reviews.apache.org/r/63133/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63133: HIVE-17054

2017-10-19 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63133/#review188704
---




ql/src/test/queries/clientpositive/groupby_join_pushdown.q
Lines 69 (patched)


Comment says in both cases optimization didn't trigger. Incorrect comment?



ql/src/test/results/clientpositive/groupby_join_pushdown.q.out
Lines 1517 (patched)


Is this expected? Either GBy needs to be in complete mode or there needs to 
be another reducer following this.



ql/src/test/results/clientpositive/groupby_join_pushdown.q.out
Lines 1632 (patched)


Is this expected? Either GBy needs to be in complete mode or there needs to 
be another reducer following this.


- Ashutosh Chauhan


On Oct. 19, 2017, 12:52 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63133/
> ---
> 
> (Updated Oct. 19, 2017, 12:52 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17054
> https://issues.apache.org/jira/browse/HIVE-17054
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17054
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 044bba1f71199cbbbd996c7e88e1eed1c0a15c21 
>   ql/src/test/queries/clientpositive/groupby_join_pushdown.q 
> a6e25685561e022db8b35e484801d874b22a03ce 
>   ql/src/test/results/clientpositive/groupby_join_pushdown.q.out 
> 95d499afef65271edbd3418b67f95691639208a2 
> 
> 
> Diff: https://reviews.apache.org/r/63133/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63124: HIVE-16603

2017-10-19 Thread Jesús Camacho Rodríguez 


> On Oct. 19, 2017, 5:17 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 3955-3956 (original), 4003-4004 (patched)
> > 
> >
> > These calls should be done outside of for() loop, else its called 
> > multiple times for each FK.

The calls are done only once per FK declaration. List can 
contain multiple FK declarations, but observe that there is an inner loop with 
the same counter (L4019).


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63124/#review188700
---


On Oct. 18, 2017, 8:59 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63124/
> ---
> 
> (Updated Oct. 18, 2017, 8:59 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16603
> https://issues.apache.org/jira/browse/HIVE-16603
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16603
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientnegative/alter_table_constraint_invalid_ref.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_pk_same_tab.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_uk_same_tab.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_wrong_ref.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_wrong_ref2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/create_with_constraints.q 
> 9700b5cd09e6578bbfedadbb0fc58c19a320c616 
>   ql/src/test/results/clientnegative/alter_table_constraint_invalid_ref.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_pk_same_tab.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_uk_same_tab.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_wrong_ref.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_wrong_ref2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/create_with_constraints.q.out 
> a675a0e6d51741505fe58680cd4a3675669b9c53 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63124/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63124: HIVE-16603

2017-10-19 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63124/#review188701
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 3955-3956 (original), 4003-4004 (patched)


The calls are done only once per FK declaration. List can 
contain multiple FK declarations, but observe that there is an inner loop with 
the same counter (L4019).


- Jesús Camacho Rodríguez


On Oct. 18, 2017, 8:59 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63124/
> ---
> 
> (Updated Oct. 18, 2017, 8:59 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16603
> https://issues.apache.org/jira/browse/HIVE-16603
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16603
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientnegative/alter_table_constraint_invalid_ref.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_pk_same_tab.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_uk_same_tab.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_wrong_ref.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_wrong_ref2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/create_with_constraints.q 
> 9700b5cd09e6578bbfedadbb0fc58c19a320c616 
>   ql/src/test/results/clientnegative/alter_table_constraint_invalid_ref.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_pk_same_tab.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_uk_same_tab.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_wrong_ref.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_wrong_ref2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/create_with_constraints.q.out 
> a675a0e6d51741505fe58680cd4a3675669b9c53 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63124/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63124: HIVE-16603

2017-10-19 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63124/#review188700
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 3955-3956 (original), 4003-4004 (patched)


These calls should be done outside of for() loop, else its called multiple 
times for each FK.


- Ashutosh Chauhan


On Oct. 18, 2017, 8:59 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63124/
> ---
> 
> (Updated Oct. 18, 2017, 8:59 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16603
> https://issues.apache.org/jira/browse/HIVE-16603
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16603
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientnegative/alter_table_constraint_invalid_ref.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_pk_same_tab.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_uk_same_tab.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_wrong_ref.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/create_with_fk_wrong_ref2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/create_with_constraints.q 
> 9700b5cd09e6578bbfedadbb0fc58c19a320c616 
>   ql/src/test/results/clientnegative/alter_table_constraint_invalid_ref.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_pk_same_tab.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_uk_same_tab.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_wrong_ref.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/create_with_fk_wrong_ref2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/create_with_constraints.q.out 
> a675a0e6d51741505fe58680cd4a3675669b9c53 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63124/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

[jira] [Created] (HIVE-17847) Exclude net.hydromatic:aggdesigner-algorithm jar as compile and runtime dependency

2017-10-19 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-17847:
---

 Summary: Exclude net.hydromatic:aggdesigner-algorithm jar as 
compile and runtime dependency
 Key: HIVE-17847
 URL: https://issues.apache.org/jira/browse/HIVE-17847
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Hive doesn't use this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17846) Fix simple groupping sets case

2017-10-19 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17846:
---

 Summary: Fix simple groupping sets case
 Key: HIVE-17846
 URL: https://issues.apache.org/jira/browse/HIVE-17846
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich


I've had a few new cases locally for HIVE-17617 and currently this query runs 
into exception.
{code}
create table tx1 (a integer,b integer,c integer);

 select  sum(c),
grouping(b),
 'no rows' as expected
fromtx1
 where   a<0
 group by b grouping sets (b);
{code}

Apparently the calcite upgrade(HIVE-17672) have broken it.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63144: HIVE-16748: Integreate YETUS to Pre-Commit

2017-10-19 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63144/#review188676
---




testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java
Lines 72-85 (patched)


I agree that this could go into a shell script generated by a velocity 
template. It would also make this a bit more testable.


- Barna Zsombor Klara


On okt. 19, 2017, 12:40 du, Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63144/
> ---
> 
> (Updated okt. 19, 2017, 12:40 du)
> 
> 
> Review request for hive, Peter Vary and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16748
> https://issues.apache.org/jira/browse/HIVE-16748
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> We already have Yetus check script ready for run in dev-support. We should 
> integrate this with the automated ptest infrastructure so that during 
> Precommit test we get a Yetus result back as Jira comment
> 
> 
> Diffs
> -
> 
>   dev-support/hive-personality.sh f3247aac6284b8dd863691b4819a10c3a896d50c 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 65a8216f6a076b0ee7baee11ca557f5e9f746316 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java
>  PRE-CREATION 
>   testutils/ptest2/src/main/resources/source-prep.vm 
> 7ad50248af02dfaeb6524a61d4895f1a8efba211 
> 
> 
> Diff: https://reviews.apache.org/r/63144/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on Cloudera sponsored sandbox ptest server.
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 63144: HIVE-16748: Integreate YETUS to Pre-Commit

2017-10-19 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63144/#review188672
---



Thanks for the patch Adam!
Mostly looks good to me. Only some minor comments.


testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java
Lines 72-85 (patched)


nit: Is it possible to write this "nicer" :) It took me a while to 
understand this. Which parameter went where, which are the actual shell 
commands.
Maybe one of the templates we used in ptest? Any idea is welcome :)



testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java
Lines 94 (patched)


It might be a good idea to print out the stack trace too


- Peter Vary


On Oct. 19, 2017, 12:40 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63144/
> ---
> 
> (Updated Oct. 19, 2017, 12:40 p.m.)
> 
> 
> Review request for hive, Peter Vary and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16748
> https://issues.apache.org/jira/browse/HIVE-16748
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> We already have Yetus check script ready for run in dev-support. We should 
> integrate this with the automated ptest infrastructure so that during 
> Precommit test we get a Yetus result back as Jira comment
> 
> 
> Diffs
> -
> 
>   dev-support/hive-personality.sh f3247aac6284b8dd863691b4819a10c3a896d50c 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 65a8216f6a076b0ee7baee11ca557f5e9f746316 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java
>  PRE-CREATION 
>   testutils/ptest2/src/main/resources/source-prep.vm 
> 7ad50248af02dfaeb6524a61d4895f1a8efba211 
> 
> 
> Diff: https://reviews.apache.org/r/63144/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on Cloudera sponsored sandbox ptest server.
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Review Request 63144: HIVE-16748: Integreate YETUS to Pre-Commit

2017-10-19 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63144/
---

Review request for hive, Peter Vary and Barna Zsombor Klara.


Bugs: HIVE-16748
https://issues.apache.org/jira/browse/HIVE-16748


Repository: hive-git


Description
---

We already have Yetus check script ready for run in dev-support. We should 
integrate this with the automated ptest infrastructure so that during Precommit 
test we get a Yetus result back as Jira comment


Diffs
-

  dev-support/hive-personality.sh f3247aac6284b8dd863691b4819a10c3a896d50c 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
65a8216f6a076b0ee7baee11ca557f5e9f746316 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java 
PRE-CREATION 
  testutils/ptest2/src/main/resources/source-prep.vm 
7ad50248af02dfaeb6524a61d4895f1a8efba211 


Diff: https://reviews.apache.org/r/63144/diff/1/


Testing
---

Tested on Cloudera sponsored sandbox ptest server.


Thanks,

Adam Szita

[jira] [Created] (HIVE-17845) insert fails if target table columns are not lowercase

2017-10-19 Thread Naresh P R (JIRA) 
Naresh P R created HIVE-17845:
-

 Summary: insert fails if target table columns are not lowercase
 Key: HIVE-17845
 URL: https://issues.apache.org/jira/browse/HIVE-17845
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Naresh P R
Assignee: Naresh P R
Priority: Minor
 Fix For: 2.3.0


eg., 
INSERT INTO TABLE EMP(ID,NAME) select * FROM SRC;
FAILED: SemanticException 1:27 '[ID,NAME]' in insert schema specification are 
not found among regular columns of default.EMP nor dynamic partition columns.. 
Error encountered near token 'NAME'

Whereas below insert is successful:
INSERT INTO TABLE EMP(id,name) select * FROM SRC;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17844) Hive JDBC driver converts floats to doubles, changing values slightly

2017-10-19 Thread Zoltan Ivanfi (JIRA) 
Zoltan Ivanfi created HIVE-17844:


 Summary: Hive JDBC driver converts floats to doubles, changing 
values slightly
 Key: HIVE-17844
 URL: https://issues.apache.org/jira/browse/HIVE-17844
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Zoltan Ivanfi


When querying data through Hive's JDBC driver, float values are automatically 
converted to doubles and will slightly differ from values seen when directly 
querying the data using a textual SQL interface.

Please note that by slight difference I don't only mean additional zeroes at 
the end, but an actual difference in the value itself. This is a result of the 
float to double conversion, as discussed 
[here|https://stackoverflow.com/questions/17504833/why-converting-from-float-to-double-changes-the-value],
 for example.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17843) UINT32 Parquet columns are handled as signed INT32-s, silently reading incorrect data

2017-10-19 Thread Zoltan Ivanfi (JIRA) 
Zoltan Ivanfi created HIVE-17843:


 Summary: UINT32 Parquet columns are handled as signed INT32-s, 
silently reading incorrect data
 Key: HIVE-17843
 URL: https://issues.apache.org/jira/browse/HIVE-17843
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Ivanfi


An unsigned 32 bit Parquet column, such as

{noformat}
optional int32 uint_32_col (UINT_32)
{noformat}

is read by Hive as if it were signed, leading to incorrect results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17842) Run checkstyle on ptest2 module with proper configuration

2017-10-19 Thread Adam Szita (JIRA) 
Adam Szita created HIVE-17842:
-

 Summary: Run checkstyle on ptest2 module with proper configuration
 Key: HIVE-17842
 URL: https://issues.apache.org/jira/browse/HIVE-17842
 Project: Hive
  Issue Type: Sub-task
Reporter: Adam Szita
Assignee: Adam Szita


Maven module ptest2 is not connected to Hive root pom, therefore if someone (or 
an automated Yetus check) runs {{mvn checkstyle}} it will not consider 
Hive-specific checkstyle settings (e.g. validates row lengths against 80, not 
100)

We need to make sure ptest2 pom has the proper checkstyle configuration



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)