[jira] [Created] (HIVE-18770) Additional tests and fixes for materialized view rewriting

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18770:
--

 Summary: Additional tests and fixes for materialized view rewriting
 Key: HIVE-18770
 URL: https://issues.apache.org/jira/browse/HIVE-18770
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


This patch requires upgrading to Calcite 1.16.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18769) Vectorization: Disable vectorization of key-less outer joins

[Gopal V (JIRA)]

Gopal V created HIVE-18769:
--

 Summary: Vectorization: Disable vectorization of key-less outer 
joins
 Key: HIVE-18769
 URL: https://issues.apache.org/jira/browse/HIVE-18769
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 3.0.0
Reporter: Gopal V


Left & Right outer joins without keys are valid in SQL and they have different 
semantics from cross-products

{code}
create temporary table foo(x int) stored as orc;
insert into foo values(1),(2);
create temporary table bar(y int) stored as orc;
select count(*) from bar right outer join foo; -- = 2
select count(*) from bar, foo; -- = 0 
{code}

canSpecializeMapJoin should bail on these cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 65746: HIVE-18768: Use Datanucleus to serialize notification updates

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65746/
---

Review request for hive, Aihua Xu, anishek, Andrew Sherman, Janaki Lahorani, 
Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.


Bugs: HIVE-18768
https://issues.apache.org/jira/browse/HIVE-18768


Repository: hive-git


Description
---

HIVE-18768: Use Datanucleus to serialize notification updates


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 7b44df41286d0356fda97914ce7460fc38518b9f 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 f9bd49574a75c813346d787767126902c192d48b 


Diff: https://reviews.apache.org/r/65746/diff/1/


Testing
---


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-18768) Use Datanucleus to serialize notification updates

[Alexander Kolbasov (JIRA)]

Alexander Kolbasov created HIVE-18768:
-

 Summary: Use Datanucleus to serialize notification updates
 Key: HIVE-18768
 URL: https://issues.apache.org/jira/browse/HIVE-18768
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.0.2, 3.0.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


HIVE-16886 added code to serialize notification updates using LOCK FOR UPDATE. 
It turns out that there is a simpler way - see HIVE-18526. The goal of this 
JIRA is to use the approach from HIVE-18526 - Datanucleus based solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65745: HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround is buggy.

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65745/
---

(Updated Feb. 22, 2018, 3:44 a.m.)


Review request for hive, Andrew Sherman, Janaki Lahorani, Zoltan Haindrich, 
Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.


Changes
---

Fixed checkstyle issue
Removed extra tbl.setParameters() call


Bugs: HIVE-18743
https://issues.apache.org/jira/browse/HIVE-18743


Repository: hive-git


Description
---

HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS 
workaround is buggy.


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 89354a2d34249903a9ff13c4ed913a68de93057e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47de215a23afb0d2589f129c75000e36c334888c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 50f873a013a9aa3cea0a2af8146484b9387c08f2 


Diff: https://reviews.apache.org/r/65745/diff/3/

Changes: https://reviews.apache.org/r/65745/diff/2-3/


Testing
---


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-18767) Some alterPartitions throw NumberFormatException: null

[Yuming Wang (JIRA)]

Yuming Wang created HIVE-18767:
--

 Summary: Some alterPartitions throw NumberFormatException: null
 Key: HIVE-18767
 URL: https://issues.apache.org/jira/browse/HIVE-18767
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.2
Reporter: Yuming Wang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65745: HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround is buggy.

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65745/#review198090
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
Line 641 (original), 632 (patched)


Interesting - looks like this line isn't needed since we know that 
parameters are present!


- Alexander Kolbasov


On Feb. 22, 2018, 2:24 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65745/
> ---
> 
> (Updated Feb. 22, 2018, 2:24 a.m.)
> 
> 
> Review request for hive, Andrew Sherman, Janaki Lahorani, Zoltan Haindrich, 
> Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18743
> https://issues.apache.org/jira/browse/HIVE-18743
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18743: CREATE TABLE on S3 data can be extremely slow. 
> DO_NOT_UPDATE_STATS workaround is buggy.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  89354a2d34249903a9ff13c4ed913a68de93057e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215a23afb0d2589f129c75000e36c334888c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013a9aa3cea0a2af8146484b9387c08f2 
> 
> 
> Diff: https://reviews.apache.org/r/65745/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 65745: HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround is buggy.

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65745/
---

(Updated Feb. 22, 2018, 2:24 a.m.)


Review request for hive, Andrew Sherman, Janaki Lahorani, Zoltan Haindrich, 
Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.


Changes
---

Avoid calculating stats for partitioned tables


Bugs: HIVE-18743
https://issues.apache.org/jira/browse/HIVE-18743


Repository: hive-git


Description
---

HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS 
workaround is buggy.


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 89354a2d34249903a9ff13c4ed913a68de93057e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47de215a23afb0d2589f129c75000e36c334888c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 50f873a013a9aa3cea0a2af8146484b9387c08f2 


Diff: https://reviews.apache.org/r/65745/diff/2/

Changes: https://reviews.apache.org/r/65745/diff/1-2/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 65663: HIVE-18710 extend inheritPerms to ACID in Hive 2.X

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65663/
---

(Updated Feb. 22, 2018, 2:22 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 2b7a57bb36 
  common/src/java/org/apache/hadoop/hive/io/HdfsUtils.java 16fc96eae9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
6bba057e76 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java a9d03d060a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 5cf2c2bc48 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 614c29bb15 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java eefa8f756c 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
5deec4b0f9 


Diff: https://reviews.apache.org/r/65663/diff/3/

Changes: https://reviews.apache.org/r/65663/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 65745: HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround is buggy.

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65745/
---

Review request for hive, Andrew Sherman, Janaki Lahorani, Zoltan Haindrich, 
Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.


Bugs: HIVE-18743
https://issues.apache.org/jira/browse/HIVE-18743


Repository: hive-git


Description
---

HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS 
workaround is buggy.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 89354a2d34249903a9ff13c4ed913a68de93057e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47de215a23afb0d2589f129c75000e36c334888c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 50f873a013a9aa3cea0a2af8146484b9387c08f2 


Diff: https://reviews.apache.org/r/65745/diff/1/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 65663: HIVE-18710 extend inheritPerms to ACID in Hive 2.X

[Sergey Shelukhin]

> On Feb. 21, 2018, 1:41 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
> > Lines 218 (patched)
> > 
> >
> > This is a short lived temp dir, there is no need to inherit perms for 
> > this. Actual partition dir perms inherit will be handled in move task.

Sorry, had MM tables in mind, this doesn't apply in 2.x


> On Feb. 21, 2018, 1:41 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 2927 (patched)
> > 
> >
> > Please see the comment above in this method. This is not required.

See the previous review iteration where I left an open item. I am not sure how 
that comment applies :)


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65663/#review197837
---


On Feb. 20, 2018, 8:33 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65663/
> ---
> 
> (Updated Feb. 20, 2018, 8:33 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 2b7a57bb36 
>   common/src/java/org/apache/hadoop/hive/io/HdfsUtils.java 16fc96eae9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 6bba057e76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java a9d03d060a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 0282763587 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 5cf2c2bc48 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 614c29bb15 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java eefa8f756c 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 5deec4b0f9 
> 
> 
> Diff: https://reviews.apache.org/r/65663/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18766) Race condition during shutdown of RemoteDriver, error messages aren't always sent

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18766:
---

 Summary: Race condition during shutdown of RemoteDriver, error 
messages aren't always sent
 Key: HIVE-18766
 URL: https://issues.apache.org/jira/browse/HIVE-18766
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


When we send an error during shutdown of the {{RemoteDriver}}, we don't wait 
for the error message to be sent. We just send it and then close the RPC 
channel. For a graceful shutdown, that doesn't seem ideal. We should at least 
wait a bit for the RPC message to be sent before shutting things down. I ran 
some tests locally and its pretty easy to hit a situation where the error 
message doesn't even get sent to the {{SparkClientImpl}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18765) SparkClientImpl swallows exception messages from the RemoteDriver

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18765:
---

 Summary: SparkClientImpl swallows exception messages from the 
RemoteDriver
 Key: HIVE-18765
 URL: https://issues.apache.org/jira/browse/HIVE-18765
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


{{SparkClientImpl#handle(ChannelHandlerContext, Error)}} swallows the cause of 
the error message:

{code}
LOG.warn("Error reported from remote driver.", msg.cause);
{code}

There should be a '{}' in the message. Without it the {{msg.cause}} info gets 
swallowed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18764) ELAPSED_TIME resource plan setting is not getting honored

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-18764:


 Summary: ELAPSED_TIME resource plan setting is not getting honored
 Key: HIVE-18764
 URL: https://issues.apache.org/jira/browse/HIVE-18764
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Trigger validation for ELAPSED_TIME counter should happen even if session is 
not created. Currently ELAPSED_TIME counter is populated only after session 
creation but a query can be waiting to get a session for a long time by the 
time trigger might have been violated. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review197794
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
Lines 65 (patched)


key = String.join(delimit, partVals);



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 311 (original), 215 (patched)


This is not introduced in this patch, but getting columns for table and 
apply to partition will not work for schema revolution. We shall get columns 
for every individual partition.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 800 (original), 632 (patched)


I don't remember but why this is get() not getUnsafe()? It sounds the same 
as getAllTables etc. Also apply to getDatabases, alterDatabase, dropDatabase, 
getDatabase and createDatabase


- Daniel Dai


On Feb. 13, 2018, 12:08 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated Feb. 13, 2018, 12:08 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  78b26374f2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d58ed677f3 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  e4e7d4239d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  80aa3bcdb4 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  9100c73beb 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86e72d8d76 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  bd61df654a 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-18762) Extend the current ownership support for tables

[kalyan kumar kalvagadda (JIRA)]

kalyan kumar kalvagadda created HIVE-18762:
--

 Summary: Extend the current ownership support for tables
 Key: HIVE-18762
 URL: https://issues.apache.org/jira/browse/HIVE-18762
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: kalyan kumar kalvagadda


Currently only a user can be a owner of hive table. It should be extended so 
that either user/role can be set a owner of a table.

With this support ownership of a table can be transferred to either user or role

I think, this is already available for hive databases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18763) LLAP IO for text should take table serde into account

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18763:
---

 Summary: LLAP IO for text should take table serde into account
 Key: HIVE-18763
 URL: https://issues.apache.org/jira/browse/HIVE-18763
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


It's only using partition SerDe right now. We should reconcile both for when 
there are changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

FINAL REMINDER: CFP for Apache EU Roadshow Closes 25th February

[Sharan F]

Hello Apache Supporters and Enthusiasts

This is your FINAL reminder that the Call for Papers (CFP) for the 
Apache EU Roadshow is closing soon. Our Apache EU Roadshow will focus on 
Cloud, IoT, Apache Tomcat, Apache Http and will run from 13-14 June 2018 
in Berlin.
Note that the CFP deadline has been extended to *25*^*th* *February *and 
it will be your final opportunity to submit a talk for thisevent.


Please make your submissions at http://apachecon.com/euroadshow18/

Also note that early bird ticket registrations to attend FOSS Backstage 
including the Apache EU Roadshow, have also been extended and will be 
available until 23^rd February. Please register at 
https://foss-backstage.de/tickets


We look forward to seeing you in Berlin!

Thanks
Sharan Foga, VP Apache Community Development

PLEASE NOTE: You are receiving this message because you are subscribed 
to a user@ or dev@ list of one or more Apache Software Foundation projects.

Re: Review Request 65716: HIVE-18696: The partition folders might not get cleaned up properly in the HiveMetaStore.add_partitions_core method if an exception occurs

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65716/#review197829
---



Would be good to know which Hive queries invoke this method.


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 3032 (original), 3065 (patched)


this code looks very similar to the block above. I know its was never the 
intention of this JIRA to do any re-factoring, but how difficult would it be to 
move all this code into a common method so that we don't have to fix the bug in 
two places? not a blocking issue though



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 3149-3150 (patched)


curious what behavior you were seeing, wondering why cancelling or 
interrupting the `Future`s doesn't work


- Sahil Takiar


On Feb. 20, 2018, 5:03 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65716/
> ---
> 
> (Updated Feb. 20, 2018, 5:03 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18696
> https://issues.apache.org/jira/browse/HIVE-18696
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The idea behind the patch is
> 
> 1) Separate the partition validation from starting the tasks which create the 
> partition folders. 
> Instead of doing the checks on the partitions and submit the tasks in one 
> loop, separated the validation into a different loop. So first iterate 
> through the partitions, validate the table/db names, and check for 
> duplicates. Then if all partitions were correct, in the second loop submit 
> the tasks to create the partition folders. This way if one of the partitions 
> is incorrect, the exception will be thrown in the first loop, before the 
> tasks are submitted. So we can be sure that no partition folder will be 
> created if the list contains an invalid partition.
> 
> 2) Handle the exceptions which occur during the execution of the tasks 
> differently.
> Previously if an exception occured in one task, the remaining tasks were 
> canceled, and the newly created partition folders were cleaned up in the 
> finally part. The problem was that it could happen that some tasks were still 
> not finished with the folder creation when cleaning up the others, so there 
> could have been leftover folders. After doing some testing it turned out that 
> this use case cannot be avoided completely when canceling the tasks.
> The idea of this patch is to set a flag if an exception is thrown in one of 
> the tasks. This flag is visible in the tasks and if its value is true, the 
> partition folders won't be created. Then iterate through the remaining tasks 
> and wait for them to finish. The tasks which are started before the flag got 
> set will then finish creating the partition folders. The tasks which are 
> started after the flag got set, won't create the partition folders, to avoid 
> unnecessary work. This way it is sure that all tasks are finished, when 
> entering the finally part where the partition folders are cleaned up.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  f483ca8 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
>  919ba78 
> 
> 
> Diff: https://reviews.apache.org/r/65716/diff/1/
> 
> 
> Testing
> ---
> 
> Added some new tests cases to the TestAddPartitions and 
> TestAddPartitionsFromPartSpec tests.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65716: HIVE-18696: The partition folders might not get cleaned up properly in the HiveMetaStore.add_partitions_core method if an exception occurs

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65716/#review197843
---



Thanks for the patch Marta!
Mostly just questions.
Peter


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 2830 (patched)


nit: formatting



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 2856 (patched)


Question: Do this has to be AtomicBoolean instead of simple boolean?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 3098 (patched)


Same as above



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Line 840 (original), 858 (patched)


Is this line only changed in formatting?



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
Lines 626 (patched)


Why is this change? Is this an incompatible change?


- Peter Vary


On Feb. 20, 2018, 5:03 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65716/
> ---
> 
> (Updated Feb. 20, 2018, 5:03 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18696
> https://issues.apache.org/jira/browse/HIVE-18696
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The idea behind the patch is
> 
> 1) Separate the partition validation from starting the tasks which create the 
> partition folders. 
> Instead of doing the checks on the partitions and submit the tasks in one 
> loop, separated the validation into a different loop. So first iterate 
> through the partitions, validate the table/db names, and check for 
> duplicates. Then if all partitions were correct, in the second loop submit 
> the tasks to create the partition folders. This way if one of the partitions 
> is incorrect, the exception will be thrown in the first loop, before the 
> tasks are submitted. So we can be sure that no partition folder will be 
> created if the list contains an invalid partition.
> 
> 2) Handle the exceptions which occur during the execution of the tasks 
> differently.
> Previously if an exception occured in one task, the remaining tasks were 
> canceled, and the newly created partition folders were cleaned up in the 
> finally part. The problem was that it could happen that some tasks were still 
> not finished with the folder creation when cleaning up the others, so there 
> could have been leftover folders. After doing some testing it turned out that 
> this use case cannot be avoided completely when canceling the tasks.
> The idea of this patch is to set a flag if an exception is thrown in one of 
> the tasks. This flag is visible in the tasks and if its value is true, the 
> partition folders won't be created. Then iterate through the remaining tasks 
> and wait for them to finish. The tasks which are started before the flag got 
> set will then finish creating the partition folders. The tasks which are 
> started after the flag got set, won't create the partition folders, to avoid 
> unnecessary work. This way it is sure that all tasks are finished, when 
> entering the finally part where the partition folders are cleaned up.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  f483ca8 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
>  919ba78 
> 
> 
> Diff: https://reviews.apache.org/r/65716/diff/1/
> 
> 
> Testing
> ---
> 
> Added some new tests cases to the TestAddPartitions and 
> TestAddPartitionsFromPartSpec tests.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65731: HIVE-18699: Check for duplicate partitions in HiveMetastore.exchange_partitions

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65731/#review197848
---



Thanks Marta!
Just a quick querstion.


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 3370 (patched)


How "expensive" is this call? Is this a simple query? What happens if the 
destintaion table has 1m partitions? :)


- Peter Vary


On Feb. 21, 2018, 11:37 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65731/
> ---
> 
> (Updated Feb. 21, 2018, 11:37 a.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18699
> https://issues.apache.org/jira/browse/HIVE-18699
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the HiveMetastore.exchange_partitions method to check if the 
> partitions to be exchanged don't exist in the dest table. If one of the 
> partitions already exists, throw a MetaException with a proper error message.
> 
> Previously an exception like this (wrapped in a MetaException) was thrown:
> Insert of object
> "org.apache.hadoop.hive.metastore.model.MPartition@4e78fff5" using statement 
> "INSERT INTO PARTITIONS
> (PART_ID,CREATE_TIME,LAST_ACCESS_TIME,PART_NAME,SD_ID,TBL_ID) VALUES 
> (?,?,?,?,?,?)" failed : The statement was
> aborted because it would have caused a duplicate key value in a unique or 
> primary key constraint or unique index
> identified by 'UNIQUEPARTITION' defined on 'PARTITIONS'.
> 
> From user point of view, the type of the exception is not changed 
> (MetaException), just the error message is changed to a more understandable 
> one.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
>  3a06aec 
> 
> 
> Diff: https://reviews.apache.org/r/65731/diff/1/
> 
> 
> Testing
> ---
> 
> Tests already exist for this use case in TestExchangePartitions:
> - testExchangePartitionsPartAlreadyExists
> - testExchangePartitionPartAlreadyExists
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65730: HIVE-18697: The HiveMetastore.exchange_partitions method throws FileNotFoundException if the given partition doesn't exist in the source table

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65730/#review197847
---


Ship it!




Ship It!

- Peter Vary


On Feb. 21, 2018, 10:36 a.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65730/
> ---
> 
> (Updated Feb. 21, 2018, 10:36 a.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18697
> https://issues.apache.org/jira/browse/HIVE-18697
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the HiveMetastore.exchange_partitions method to check if the 
> partitionsToExchange list is empty and if it is throw a MetaException with a 
> proper error message that no partition exists with the given values for the 
> source table. Previously a FileNotFoundException was thrown (wrapped in a 
> MetaException) when tried to move the partition folder to the dest table. So 
> the type of the exception was not changed, only the error message.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
>  3a06aec 
> 
> 
> Diff: https://reviews.apache.org/r/65730/diff/1/
> 
> 
> Testing
> ---
> 
> There are tests for this use case in TestExchangePartitions:
> - testExchangePartitionsNoPartExists
> - testExchangePartitionNoPartExists
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Review Request 65731: HIVE-18699: Check for duplicate partitions in HiveMetastore.exchange_partitions

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65731/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18699
https://issues.apache.org/jira/browse/HIVE-18699


Repository: hive-git


Description
---

Extended the HiveMetastore.exchange_partitions method to check if the 
partitions to be exchanged don't exist in the dest table. If one of the 
partitions already exists, throw a MetaException with a proper error message.

Previously an exception like this (wrapped in a MetaException) was thrown:
Insert of object
"org.apache.hadoop.hive.metastore.model.MPartition@4e78fff5" using statement 
"INSERT INTO PARTITIONS
(PART_ID,CREATE_TIME,LAST_ACCESS_TIME,PART_NAME,SD_ID,TBL_ID) VALUES 
(?,?,?,?,?,?)" failed : The statement was
aborted because it would have caused a duplicate key value in a unique or 
primary key constraint or unique index
identified by 'UNIQUEPARTITION' defined on 'PARTITIONS'.

>From user point of view, the type of the exception is not changed 
>(MetaException), just the error message is changed to a more understandable 
>one.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47de215 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
 3a06aec 


Diff: https://reviews.apache.org/r/65731/diff/1/


Testing
---

Tests already exist for this use case in TestExchangePartitions:
- testExchangePartitionsPartAlreadyExists
- testExchangePartitionPartAlreadyExists


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-18761) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.RuntimeException: java.lang.NullPointerException

[chandhirakumar (JIRA)]

chandhirakumar created HIVE-18761:
-

 Summary: FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. 
MetaException(message:java.lang.RuntimeException: java.lang.NullPointerException
 Key: HIVE-18761
 URL: https://issues.apache.org/jira/browse/HIVE-18761
 Project: Hive
  Issue Type: Bug
Reporter: chandhirakumar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18760) ExchangeHive.com - Domain Name Inquiry.

[Craig Owens (JIRA)]

Craig Owens created HIVE-18760:
--

 Summary: ExchangeHive.com - Domain Name Inquiry.
 Key: HIVE-18760
 URL: https://issues.apache.org/jira/browse/HIVE-18760
 Project: Hive
  Issue Type: Wish
  Components: Website
Reporter: Craig Owens


*ExchangeHive.com - Domain Name Inquiry*.

Hello,

 

I have a domain name which I think will interest you. The name is 
ExchangeHive.com.

It is great for the market you are in. This is because the name itself will 
give you the edge on your competitors. Finding a quality, keyword rich domain 
like this one, applicable to your business, is not an easy task, as most of the 
great .com's are already in use.

Domain ownership transfer is free and painless *ExchangeHive.com - Domain Name 
Inquiry* and you can expect a same day ownership transfer. 

 

Thank you for your time, I am available to answer any questions you may have by 
email or telephone.

If you are interested and would like to make an offer please reply to this 
email. I will consider all reasonable offers.

 

Kindly forward this message to the appropriate authority in your organisation 
for consideration.

 

Sincerely Craig Email: *salesdomai...@hotmail.com*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 65730: HIVE-18697: The HiveMetastore.exchange_partitions method throws FileNotFoundException if the given partition doesn't exist in the source table

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65730/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18697
https://issues.apache.org/jira/browse/HIVE-18697


Repository: hive-git


Description
---

Extended the HiveMetastore.exchange_partitions method to check if the 
partitionsToExchange list is empty and if it is throw a MetaException with a 
proper error message that no partition exists with the given values for the 
source table. Previously a FileNotFoundException was thrown (wrapped in a 
MetaException) when tried to move the partition folder to the dest table. So 
the type of the exception was not changed, only the error message.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 47de215 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
 3a06aec 


Diff: https://reviews.apache.org/r/65730/diff/1/


Testing
---

There are tests for this use case in TestExchangePartitions:
- testExchangePartitionsNoPartExists
- testExchangePartitionNoPartExists


Thanks,

Marta Kuczora