Re: Review Request 68827: HIVE-20545 : Exclude large-sized parameters from serialization of Table and Partition thrift objects in HMS notifications

2018-10-01 Thread Bharathkrishna Guruvayoor Murali via Review Board


> On Sept. 26, 2018, 11:20 p.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 918 (patched)
> > 
> >
> > You already have this check when you call this function.
> 
> Bharathkrishna Guruvayoor Murali wrote:
> I thought this could be used separately as a utility method, if the user 
> has just one predicate. Should I just keep the other method which accepts a 
> list of predicates and make this one private?

Closing this issue. Maintaining both methods as they can be used as separate 
utilities if needed.


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68827/#review209055
---


On Oct. 2, 2018, 5:19 a.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68827/
> ---
> 
> (Updated Oct. 2, 2018, 5:19 a.m.)
> 
> 
> Review request for hive and Alexander Kolbasov.
> 
> 
> Bugs: HIVE-20545
> https://issues.apache.org/jira/browse/HIVE-20545
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Clients can add large-sized parameters in Table/Partition objects. So we need 
> to enable adding regex patterns through HiveConf to match parameters to be 
> filtered from table and partition objects before serialization in HMS 
> notifications.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  30ea7f81292b0db54f4eb82468191fda38f9a0d4 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  c681a87a1c6b10a4f9494e49a42282cf90027ad7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java
>  7ff168f7931f91fe17f7d38df848ba2eed33c463 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java
>  2668b053205f48226da442ce65fcc2d7f6e76763 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  30de1c4cfa1cf019186b10583a06da0bf5491634 
> 
> 
> Diff: https://reviews.apache.org/r/68827/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>



Re: Review Request 68827: HIVE-20545 : Exclude large-sized parameters from serialization of Table and Partition thrift objects in HMS notifications

2018-10-01 Thread Bharathkrishna Guruvayoor Murali via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68827/
---

(Updated Oct. 2, 2018, 5:19 a.m.)


Review request for hive and Alexander Kolbasov.


Changes
---

I added init() method to MessageFactory, and overriding it in 
JSONMessageFactory to initialize the static variables. Although the exception 
traces look mostly similar, I guess this adds a bit more clarity and 
information about the exception if the regex is invalid.


Bugs: HIVE-20545
https://issues.apache.org/jira/browse/HIVE-20545


Repository: hive-git


Description
---

Clients can add large-sized parameters in Table/Partition objects. So we need 
to enable adding regex patterns through HiveConf to match parameters to be 
filtered from table and partition objects before serialization in HMS 
notifications.


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 30ea7f81292b0db54f4eb82468191fda38f9a0d4 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 c681a87a1c6b10a4f9494e49a42282cf90027ad7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java
 7ff168f7931f91fe17f7d38df848ba2eed33c463 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java
 2668b053205f48226da442ce65fcc2d7f6e76763 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
 30de1c4cfa1cf019186b10583a06da0bf5491634 


Diff: https://reviews.apache.org/r/68827/diff/5/

Changes: https://reviews.apache.org/r/68827/diff/4-5/


Testing
---


Thanks,

Bharathkrishna Guruvayoor Murali



[jira] [Created] (HIVE-20667) fix bug - CREATE TABLE tabletype problem while executing addDbAndTabToOutputs

2018-10-01 Thread Maolin Tu (JIRA)
Maolin Tu created HIVE-20667:


 Summary: fix bug - CREATE TABLE tabletype problem while executing 
addDbAndTabToOutputs
 Key: HIVE-20667
 URL: https://issues.apache.org/jira/browse/HIVE-20667
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.1.0, 2.3.1
Reporter: Maolin Tu
Assignee: Maolin Tu


problem is in 
[ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java|https://github.com/apache/hive/pull/426/files#diff-f2040374293a91cbcc6594ee571b20e4]

In the release version of Hive, outputs'TableType always be MANAGED_TABLE, no 
matter the value of isExt is true of false. While I using post_execute_hook, I 
found this problem because I cannot get correct tabletype in HiveHookContext.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 68827: HIVE-20545 : Exclude large-sized parameters from serialization of Table and Partition thrift objects in HMS notifications

2018-10-01 Thread Bharathkrishna Guruvayoor Murali via Review Board


> On Sept. 26, 2018, 5:51 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java
> > Line 297 (original), 310 (patched)
> > 
> >
> > This is used by the notifications that (we think) we understand, but it 
> > is also used by JSONAcidWriteMessage. So what happens if someone uses your 
> > new mechanism to reduce the size of messages, but affects 
> > JSONAcidWriteMessage? In other words there could be multile uses for 
> > notifications in a complex system, and this mechanism affects them all.

For now, I feel this change is reasonable and when there is a specific 
requirement that it should not filter parameters from other types of 
messages(if any), it can be taken up later on as an improvement.


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68827/#review209031
---


On Sept. 30, 2018, 6:55 p.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68827/
> ---
> 
> (Updated Sept. 30, 2018, 6:55 p.m.)
> 
> 
> Review request for hive and Alexander Kolbasov.
> 
> 
> Bugs: HIVE-20545
> https://issues.apache.org/jira/browse/HIVE-20545
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Clients can add large-sized parameters in Table/Partition objects. So we need 
> to enable adding regex patterns through HiveConf to match parameters to be 
> filtered from table and partition objects before serialization in HMS 
> notifications.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  30ea7f81292b0db54f4eb82468191fda38f9a0d4 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  c681a87a1c6b10a4f9494e49a42282cf90027ad7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java
>  2668b053205f48226da442ce65fcc2d7f6e76763 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  30de1c4cfa1cf019186b10583a06da0bf5491634 
> 
> 
> Diff: https://reviews.apache.org/r/68827/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>



Re: Review Request 68868: HIVE-17043: Remove non unique columns from group by keys if not referenced later

2018-10-01 Thread Jesús Camacho Rodríguez

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68868/#review209136
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
Lines 223 (patched)


Should it be key.contains(columns) ?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
Lines 293 (patched)


Can we add a comment here? 'They should all be nullable'



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
Line 285 (original), 311 (patched)


Can we either return a Pair, or make it void and that it sets both _keys_ 
and _nonNullablekeys_? Currently it is a bit weird that one of them is set via 
return of the method, and the second one from the method.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
Line 363 (original), 414 (patched)


return...



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
Lines 53 (patched)


This should be a new metadata provider or a modification of the existing 
_RelMdUniqueKeys_ but introducing a new boolean parameter 
_acceptEstimatedResults_. However, this would need changes in Calcite side. 
Please, leave a TODO comment though.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
Lines 59 (patched)


You can make all these methods private and leave a single point of entry to 
the method (the one that has a RelNode parameter).



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
Lines 288 (patched)


We may have Volcano node too, we need to include it.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
Line 320 (original), 320 (patched)


We do not use metadata providers anymore for unique keys estimation, which 
means some features will be disabled, e.g., caching of metadata results. I am 
not sure whether this may cause an increase in compilation time for several 
queries, specially those containing many joins, but it would be worth leaving a 
comment and maybe monitoring it in future.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
Lines 331 (patched)


Why is this neeeded now? _getUniqueKeys_ did not have a handler for 
HepRelVertex, hence I would expect this would have returned null keys in any 
case. Also note that we may use other planners now, hence we may have other 
special nodes such as the Volcano nodes.


- Jesús Camacho Rodríguez


On Sept. 28, 2018, 1:23 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68868/
> ---
> 
> (Updated Sept. 28, 2018, 1:23 a.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-17043
> https://issues.apache.org/jira/browse/HIVE-17043
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch implements/test the following optimizations
> * Removal of group by on primary keys
> * Reduction of group by keys on primary keys
> * is NOT NULL filter removal if NOT NULL constraint is defined
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties def356176b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java
>  635d27e723 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 42e60de6a8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
>  f43ef01293 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
>  5857f730a8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdColumnUniqueness.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  1ca1937ed9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
>  3bf62c535c 
>   ql/src/test/queries/clientpositive/constraints_optimization.q PRE-CREATION 
>   

[jira] [Created] (HIVE-20666) HiveServer2 Interactive LLAP reconnect to already running Yarn app

2018-10-01 Thread Hari Sekhon (JIRA)
Hari Sekhon created HIVE-20666:
--

 Summary: HiveServer2 Interactive LLAP reconnect to already running 
Yarn app
 Key: HIVE-20666
 URL: https://issues.apache.org/jira/browse/HIVE-20666
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2, llap
Affects Versions: 1.2.1
Reporter: Hari Sekhon


Improve HiveServer2 Interactive LLAP to reconnect to already running hive llap 
yarn app.

Currently HiveServer2 Interactive startup may fail with the following error if 
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired 
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live 
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3', 
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput': 
True, 'user': 'hive', 'stderr': -1}{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HIVE-20665) Hive Parallel Tasks - Hive Configuration ConcurrentModificationException

2018-10-01 Thread BELUGA BEHR (JIRA)
BELUGA BEHR created HIVE-20665:
--

 Summary: Hive Parallel Tasks - Hive Configuration 
ConcurrentModificationException
 Key: HIVE-20665
 URL: https://issues.apache.org/jira/browse/HIVE-20665
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.1.0, 2.3.2, 4.0.0
Reporter: BELUGA BEHR


When parallel tasks are enabled in Hive, all of the resulting queries share the 
same Hive configuration.  This is problematic as each query will modify the 
same {{HiveConf}} object with things like query ID and query text.  This will 
overwrite each other and cause {{ConcurrentModificationException}} issues.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys

2018-10-01 Thread Saurabh Seth (JIRA)
Saurabh Seth created HIVE-20664:
---

 Summary: Potential ArrayIndexOutOfBoundsException in 
VectorizedOrcAcidRowBatchReader.findMinMaxKeys
 Key: HIVE-20664
 URL: https://issues.apache.org/jira/browse/HIVE-20664
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Saurabh Seth
Assignee: Saurabh Seth


[~ekoifman], could you please confirm if my understanding is correct and if so, 
review the fix?

In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code 
snippet that identifies the first and last stripe indices in the current split 
could result in an ArrayIndexOutOfBoundsException if a complete split is within 
the same stripe:
{noformat}
    for(int i = 0; i < stripes.size(); i++) {
  StripeInformation stripe = stripes.get(i);
  long stripeEnd = stripe.getOffset() + stripe.getLength();
  if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
firstStripeIndex = i;
  }
  if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
  stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
//the last condition is for when both splitStart and splitEnd are in
// the same stripe
lastStripeIndex = i;
  }
}
{noformat}
Consider the example where there are 2 stripes - 0-500 and 500-1000 and 
splitStart is 600 and splitEnd is 800.

In the first iteration of the loop, stripe.getOffset() is 0 and stripeEnd is 
500. In this iteration, neither of the if statement conditions will be met and 
firstSripeIndex as well as lastStripeIndex remain -1.

In the second iteration of the loop stripe.getOffset() is 500, stripeEnd is 
1000, The first if statement condition will not be met in this case because 
stripe's offset (500) is not greater than or equal to the splitStart (600). 
However, in the second if statement, splitEnd (800) is <= stripeEnd(1000) and 
it will try to compute the last condition 
{{stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset()}}. This will 
throw an ArrayIndexOutOfBoundsException because firstStripeIndex is still -1.

I'm not sure if this scenario is possible at all, hence logging this as a low 
priority issue. Perhaps block based split generation using BISplitStrategy 
could trigger this?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 68889: HIVE-20610 : TestDbNotificationListener should not use /tmp directory

2018-10-01 Thread Andrew Sherman via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68889/#review209131
---




itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
Lines 123 (patched)


General comment: don't try to do too much in static initializers in server 
code. Just like in HIVE-20545 you have to consider what will happen if there is 
a failure during initialization, and the result is always ugly. In this case it 
looks safe but IT MADE ME THINK which is bad.



itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
Line 802 (original), 804 (patched)


This looks ugly to me. I think the string concatenation operator + should 
be separated on both sides by spaces. I think that is what is most commonly 
used on Hive - I'll leave it to you to check. But the usage is here is 
different from that in the static initializer code and that inconsistency is 
ugly too. IMHO You should teach Intellij to do your formatting and then let it 
decide this stuff


- Andrew Sherman


On Oct. 1, 2018, 7:09 a.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68889/
> ---
> 
> (Updated Oct. 1, 2018, 7:09 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Andrew Sherman.
> 
> 
> Bugs: HIVE-20610
> https://issues.apache.org/jira/browse/HIVE-20610
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding java.io.tmpdir as tmp directory instead of /tmp
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  82429e36a5 
> 
> 
> Diff: https://reviews.apache.org/r/68889/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>



[jira] [Created] (HIVE-20663) Enable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]

2018-10-01 Thread Karen Coppage (JIRA)
Karen Coppage created HIVE-20663:


 Summary: Enable 
TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
 Key: HIVE-20663
 URL: https://issues.apache.org/jira/browse/HIVE-20663
 Project: Hive
  Issue Type: Test
Reporter: Karen Coppage


results/clientpositive/llap/load_dyn_part3.q.out currently requires 

Column stats: COMPLETE

but test outputs

Column stats: PARTIAL



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (HIVE-20662) Disable TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]

2018-10-01 Thread Karen Coppage (JIRA)
Karen Coppage created HIVE-20662:


 Summary: Disable 
TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3]
 Key: HIVE-20662
 URL: https://issues.apache.org/jira/browse/HIVE-20662
 Project: Hive
  Issue Type: Bug
Reporter: Karen Coppage
Assignee: Karen Coppage


TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] is more than flaky.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 68474: HIVE-20440: Create better cache eviction policy for SmallTableCache

2018-10-01 Thread Sahil Takiar

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68474/#review209130
---




ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Line 60 (original), 69 (patched)


keep the explicit cache method and call it in `MapJoinOperator#closeOp`. 
This way when a task finishes, we still keep the small table around for at 
least 30 seconds, which gives any tasks scheduled in the future a chance to 
re-use the small table.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 75 (patched)


can u add some javadocs to this class explaining what it is doing



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 82 (patched)


rename to something like `cleanupService`



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 90 (patched)


nit: make `INTEGER_ONE` a static import



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 91 (patched)


"SmallTableCache maintenance thread" -> "SmallTableCache Cleanup Thread"



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
Lines 117 (patched)


replace with `cacheL1.get(key, valueLoader)` where `valueLoader` loads from 
`cacheL2`


- Sahil Takiar


On Sept. 19, 2018, 11:14 p.m., Antal Sinkovits wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68474/
> ---
> 
> (Updated Sept. 19, 2018, 11:14 p.m.)
> 
> 
> Review request for hive, Naveen Gangam, Sahil Takiar, Adam Szita, and Xuefu 
> Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> I've modified the SmallTableCache to use guava cache, with soft references.
> By using a value loader, I've also eliminated the synchronization on the 
> intern-ed string of the path.
> 
> 
> Diffs
> -
> 
>   ql/pom.xml d73deba440702ec39fc5610df28e0fe54baef025 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java 
> cf27e92bafdc63096ec0fa8c3106657bab52f370 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java 
> 3293100af96dc60408c53065fa89143ead98f818 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSmallTableCache.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68474/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Antal Sinkovits
> 
>



[jira] [Created] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1

2018-10-01 Thread Laszlo Pinter (JIRA)
Laszlo Pinter created HIVE-20661:


 Summary: Dynamic partitions loading calls add partition for every 
partition 1-by-1
 Key: HIVE-20661
 URL: https://issues.apache.org/jira/browse/HIVE-20661
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 4.0.0
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter


Hive.loadDynamicPartitions creates partitions using a threadpool, but the 
update of the Metastore  via the MetastoreClient is done 1-by-1. This adds 
unnecessary extra calls. The partitions should be created in one batch. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 68683: Add new configuration to set the size of the global compile lock

2018-10-01 Thread denys kuzmenko via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68683/
---

(Updated Oct. 1, 2018, 11:42 a.m.)


Review request for hive, Zoltan Haindrich, Zoltan Haindrich, Naveen Gangam, and 
Peter Vary.


Bugs: HIVE-20535
https://issues.apache.org/jira/browse/HIVE-20535


Repository: hive-git


Description
---

When removing the compile lock, it is quite risky to remove it entirely.

It would be good to provide a pool size for the concurrent compilation, so the 
administrator can limit the load


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d1e6631975 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java dad2035362 
  ql/src/java/org/apache/hadoop/hive/ql/lock/CompileLock.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/lock/CompileLockFactory.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/CompileLockTest.java PRE-CREATION 


Diff: https://reviews.apache.org/r/68683/diff/8/

Changes: https://reviews.apache.org/r/68683/diff/7-8/


Testing
---

Added CompileLockTest


File Attachments


HIVE-20535.1.patch
  
https://reviews.apache.org/media/uploaded/files/2018/09/13/41f5a84a-70e5-4882-99c1-1cf98c4364e4__HIVE-20535.1.patch
HIVE-20535.14.patch
  
https://reviews.apache.org/media/uploaded/files/2018/09/25/335b0f4b-ea94-41d4-881a-ec8bb870a376__HIVE-20535.14.patch
HIVE-20535.14.patch
  
https://reviews.apache.org/media/uploaded/files/2018/09/25/a92b6da2-eeba-46ee-9409-162653826172__HIVE-20535.14.patch
HIVE-20535.14.patch
  
https://reviews.apache.org/media/uploaded/files/2018/09/25/9db4cf76-9188-48fb-bd3d-5b28e43a791b__HIVE-20535.14.patch


Thanks,

denys kuzmenko



Review Request 68889: HIVE-20610 : TestDbNotificationListener should not use /tmp directory

2018-10-01 Thread Bharathkrishna Guruvayoor Murali via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68889/
---

Review request for hive, Alexander Kolbasov and Andrew Sherman.


Bugs: HIVE-20610
https://issues.apache.org/jira/browse/HIVE-20610


Repository: hive-git


Description
---

Adding java.io.tmpdir as tmp directory instead of /tmp


Diffs
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 82429e36a5 


Diff: https://reviews.apache.org/r/68889/diff/1/


Testing
---


Thanks,

Bharathkrishna Guruvayoor Murali