[jira] [Created] (HIVE-19146) Delete dangling q.out

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-19146:
---

 Summary: Delete dangling q.out 
 Key: HIVE-19146
 URL: https://issues.apache.org/jira/browse/HIVE-19146
 Project: Hive
  Issue Type: Test
  Components: Test
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Fails TestDanglingQOuts



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19145) Stabilize statsoptimizer.q test

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-19145:
---

 Summary: Stabilize statsoptimizer.q test
 Key: HIVE-19145
 URL: https://issues.apache.org/jira/browse/HIVE-19145
 Project: Hive
  Issue Type: Test
  Components: Test
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Uses current_date() which is prone to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19144) TestSparkCliDriver:subquery_scalar - golden file needs to be udpated

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19144:
--

 Summary: TestSparkCliDriver:subquery_scalar - golden file needs to 
be udpated
 Key: HIVE-19144
 URL: https://issues.apache.org/jira/browse/HIVE-19144
 Project: Hive
  Issue Type: Sub-task
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0
 Attachments: HIVE-19144.1.patch

Looks like HIVE-18979 missed the update



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19143) Update golden files for negative tests

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-19143:
---

 Summary: Update golden files for negative tests
 Key: HIVE-19143
 URL: https://issues.apache.org/jira/browse/HIVE-19143
 Project: Hive
  Issue Type: Test
  Components: Test
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Missed in HIVE-18859



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review200783
---


Ship it!




Ship It!

- Thejas Nair


On April 10, 2018, 12:25 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> ---
> 
> (Updated April 10, 2018, 12:25 a.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
>  6f4ec6f1ea 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  2f7fa24558 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  0bbaf7e459 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856de87 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  89b400697b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  940a1bf276 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66503/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 66368: HIVE-18609: Results cache invalidation based on table updates

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66368/
---

(Updated April 10, 2018, 12:29 a.m.)


Review request for hive, Gopal V and Jesús Camacho Rodríguez.


Changes
---

Rebase with master


Bugs: HIVE-18609
https://issues.apache.org/jira/browse/HIVE-18609


Repository: hive-git


Description
---

- Save ValidTxnWriteIdList when saving query to the results cache.
- Compare the write ID list for each transactional table during results cache 
lookup.
- Add configuration to determine if queries with non-transactional tables 
should be cached.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35378 
  itests/src/test/resources/testconfiguration.properties 28c14ebc4c 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 79db006c74 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
ac5ae573d6 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 44a7496136 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3b74abacf3 
  ql/src/test/queries/clientpositive/results_cache_1.q 4aea60e1e5 
  ql/src/test/queries/clientpositive/results_cache_2.q 96a90925f6 
  ql/src/test/queries/clientpositive/results_cache_capacity.q 9f54577009 
  ql/src/test/queries/clientpositive/results_cache_empty_result.q 621367141e 
  ql/src/test/queries/clientpositive/results_cache_invalidation.q PRE-CREATION 
  ql/src/test/queries/clientpositive/results_cache_lifetime.q 60ffe96a04 
  ql/src/test/queries/clientpositive/results_cache_quoted_identifiers.q 
4802f43ba9 
  ql/src/test/queries/clientpositive/results_cache_temptable.q 9e0de765cb 
  ql/src/test/queries/clientpositive/results_cache_transactional.q PRE-CREATION 
  ql/src/test/queries/clientpositive/results_cache_with_masking.q b4fcdd57eb 
  ql/src/test/results/clientpositive/llap/results_cache_invalidation.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/results_cache_transactional.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/results_cache_invalidation.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/results_cache_transactional.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/66368/diff/2/

Changes: https://reviews.apache.org/r/66368/diff/1-2/


Testing
---

qtests added.


Thanks,

Jason Dere

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
---

(Updated April 10, 2018, 12:25 a.m.)


Review request for hive and Thejas Nair.


Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-19126


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 6f4ec6f1ea 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 2f7fa24558 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 0bbaf7e459 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856de87 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 89b400697b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 940a1bf276 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66503/diff/3/

Changes: https://reviews.apache.org/r/66503/diff/2-3/


Testing
---


Thanks,

Vaibhav Gumashta

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review200780
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Line 308 (original), 305 (patched)


(It would be good to point out that this is based on an estimation, not the 
exact memory footprint).

The maximum memory in bytes that the cached objects can use. Memory used is 
calculated based on estimated size of tables and partitions in the cache.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 165 (patched)


I see LLAP uses HiveConf.getSizeVar(conf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE);
 to get a type of size
 can you please check if MetastoreConf.getLongVar(conf, 
ConfVars.CACHED_RAW_STORE_MAX_CACHE_MEMORY) does the conversion to bytes ?


- Thejas Nair


On April 9, 2018, 10:42 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> ---
> 
> (Updated April 9, 2018, 10:42 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
>  6f4ec6f1ea 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  2f7fa24558 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  0bbaf7e459 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856de87 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  89b400697b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  940a1bf276 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66503/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Apache Hive 3.0.0 release preparation

[Vineet Garg]

I have created an umbrella jira to investigate and fix test failures for hive 
3.0.0. LINK : https://issues.apache.org/jira/browse/HIVE-19142.
Please link any other existing jira related to test failure with this umbrella 
jira.

Also, how do we run tests on branch-3? Is there some setup to be done?

-Vineet

On Apr 9, 2018, at 4:26 AM, Zoltan Haindrich 
> wrote:

Hello

A few weeks earlier I've tried to hunt down this problem...
so...to my best knowledge the cause of this seems to be the following:

* in some cases the "cleanup" after a failed query may somehow leave some 
threads behind...
* these threads have reference to the "customized" session classloader - this 
makes the threads more memory hungry
* after a while these threads/classloaders eat up the heap...

I've opened HIVE-18522 for this thread issue

I think this problem is not new ...and it might have been present earlier as 
well...the only thing what changed is that there were a few more new features 
which have added new udfs/etc which made the memory cost of a session more 
heavier..
...and as a sidenote: I'm not convinced that this issue will arise in a proper 
hs2 setup - as it might be easily connected to the fact that these tests are 
using the cli driver to execute the tests.


cheers,
Zoltan

On 7 Apr 2018 7:15 p.m., Ashutosh Chauhan 
> wrote:
We need to investigate and find out root cause of these failures. If its
determined that its a corner case and fix is non-trivial then we may
release note it under known issues. But ideally we should fix these
failures.
Cutting a branch should make it easier since branch is expected to receive
lot less commits as compared to master so it should be faster to stabilize
branch.

On Fri, Apr 6, 2018 at 10:49 AM, Eugene Koifman 
>
wrote:

Cutting the branch before the tests are stabilized would mean we have to
fix them in 2 places.

On 4/6/18, 10:05 AM, "Thejas Nair" 
> wrote:

   That needs to be cleaned up. There are far too many right now, its
   just not handful of flaky tests.


   On Fri, Apr 6, 2018 at 2:48 AM, Peter Vary 
> wrote:
Hi Team,

I am new to the Hive release process and it is not clear to me how
the failing tests are handled. Do we plan to fix the failing tests before
release? Or it is accepted to cut a new major release with known test
issues.

Thanks,
Peter

On Apr 5, 2018, at 8:25 PM, Vineet Garg 
>
wrote:

Hello,

I plan to cut off branch for Hive 3.0.0 on Monday (9 April) since
bunch of folks have big patches pending.

Regards,
Vineet G

On Apr 2, 2018, at 3:14 PM, Vineet Garg 
>
wrote:

Hello,

We have enough votes to prepare a release candidate for Hive
3.0.0. I am going to cutoff a branch in a day or two. I’ll send an email as
soon as I have the branch ready.
Meanwhile there are approximately 69 JIRAs which are currently
opened with fix version 3.0.0. I’ll appreciate if their respective owners
would update the JIRA if it is a blocker. Otherwise I’ll update them to
defer the fix version to next release.

Regards,
Vineet G

[jira] [Created] (HIVE-19142) Umbrella: branch-3 failing tests

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19142:
--

 Summary: Umbrella: branch-3 failing tests
 Key: HIVE-19142
 URL: https://issues.apache.org/jira/browse/HIVE-19142
 Project: Hive
  Issue Type: Test
Reporter: Vineet Garg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19141) TestNegativeCliDriver insert_into_notnull_constraint, insert_into_acid_notnull failing

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19141:
--

 Summary: TestNegativeCliDriver insert_into_notnull_constraint, 
insert_into_acid_notnull failing
 Key: HIVE-19141
 URL: https://issues.apache.org/jira/browse/HIVE-19141
 Project: Hive
  Issue Type: Test
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0, 3.1.0


These tests have been consistently failing for a while. I suspect HIVE-18727 
has caused these failures. HIVE-18727 changed the code to throw ERROR instead 
of EXCEPTION if constraints are violated. I guess Negative cli driver doesn't 
handle errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
---

(Updated April 9, 2018, 10:42 p.m.)


Review request for hive and Thejas Nair.


Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-19126


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 6f4ec6f1ea 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 2f7fa24558 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 0bbaf7e459 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856de87 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 89b400697b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 940a1bf276 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66503/diff/2/

Changes: https://reviews.apache.org/r/66503/diff/1-2/


Testing
---


Thanks,

Vaibhav Gumashta

Re: Review Request 66516: HIVE-19138: Results cache: allow queries waiting on pending cache entries to check cache again if pending query fails

[Jason Dere]

> On April 9, 2018, 10:23 p.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Line 14642 (original), 14643 (patched)
> > 
> >
> > Does the loop only exit if cacheEntry is non-null?

The loop is a do .. while(false), which normally should exit after a single 
iteration. The loop should only continue to iterate in the event that 
cacheEntry.waitForValidStatus() returned false.


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66516/#review200772
---


On April 9, 2018, 9:53 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66516/
> ---
> 
> (Updated April 9, 2018, 9:53 p.m.)
> 
> 
> Review request for hive, Deepak Jaiswal and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the pending query fails, allow Hive to try to check the cache again in 
> case the cache has another cached/pending result that can be used to answer 
> the query.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 3b74abacf3 
> 
> 
> Diff: https://reviews.apache.org/r/66516/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

Re: Review Request 66516: HIVE-19138: Results cache: allow queries waiting on pending cache entries to check cache again if pending query fails

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66516/#review200772
---




ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Line 14642 (original), 14643 (patched)


Does the loop only exit if cacheEntry is non-null?


- Gopal V


On April 9, 2018, 9:53 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66516/
> ---
> 
> (Updated April 9, 2018, 9:53 p.m.)
> 
> 
> Review request for hive, Deepak Jaiswal and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the pending query fails, allow Hive to try to check the cache again in 
> case the cache has another cached/pending result that can be used to answer 
> the query.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 3b74abacf3 
> 
> 
> Diff: https://reviews.apache.org/r/66516/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

[jira] [Created] (HIVE-19140) Update metastore upgrade scripts to prepare for 3.1.0 development

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19140:
--

 Summary: Update metastore upgrade scripts to prepare for 3.1.0 
development
 Key: HIVE-19140
 URL: https://issues.apache.org/jira/browse/HIVE-19140
 Project: Hive
  Issue Type: Task
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.1.0


Now that branch for hive 3.0.0 is cut and we have started preparing for hive 
3.1.0 development we need to add metastore upgrade scripts to upgrade from 
3.0.0 to 3.1.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19139) Hive.getValidPartitionsInPath() issue

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19139:
-

 Summary: Hive.getValidPartitionsInPath() issue
 Key: HIVE-19139
 URL: https://issues.apache.org/jira/browse/HIVE-19139
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Reporter: Eugene Koifman


this method looks like this.  This will not work for multi-stmt txns since each 
statement uses the same writeId but different statementId.
{noformat}
// The non-MM path only finds new partitions, as it is looking at the temp path.
// To produce the same effect, we will find all the partitions affected by this 
txn ID.
// Note: we ignore the statement ID here, because it's currently irrelevant for 
MoveTask
// where this is used; we always want to load everything; also the only case 
where
// we have multiple statements anyway is union.
Utilities.FILE_OP_LOGGER.trace(
 "Looking for dynamic partitions in {} ({} levels)", loadPath, numDP);
Path[] leafStatus = Utilities.getMmDirectoryCandidates(
 fs, loadPath, numDP, numLB, null, writeId, -1, conf, isInsertOverwrite); 
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66516: HIVE-19138: Results cache: allow queries waiting on pending cache entries to check cache again if pending query fails

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66516/
---

Review request for hive, Deepak Jaiswal and Gopal V.


Repository: hive-git


Description
---

If the pending query fails, allow Hive to try to check the cache again in case 
the cache has another cached/pending result that can be used to answer the 
query.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3b74abacf3 


Diff: https://reviews.apache.org/r/66516/diff/1/


Testing
---


Thanks,

Jason Dere

[jira] [Created] (HIVE-19138) Results cache: allow queries waiting on pending cache entries to check cache again if pending query fails

[Jason Dere (JIRA)]

Jason Dere created HIVE-19138:
-

 Summary: Results cache: allow queries waiting on pending cache 
entries to check cache again if pending query fails
 Key: HIVE-19138
 URL: https://issues.apache.org/jira/browse/HIVE-19138
 Project: Hive
  Issue Type: Sub-task
Reporter: Jason Dere
Assignee: Jason Dere


HIVE-18846 allows the results cache to refer to currently executing queries so 
that another query can wait for these results to become ready in the results 
cache. If the pending query fails then Hive will automatically skip the cache 
and do the full query compilation. Make a fix here so that if the pending query 
fails, Hive will still try to check the cache again in case the cache has 
another cached/pending result that can be used to answer the query.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Hive 3.0.0 Release branch cutoff

[Vineet Garg]

Hello,

To keep the naming conventions for release branch I have created branch-3 for 
Apache hive 3.0.0 release. Please use this branch instead of branch-3.0.0.

Vineet G
.
On Apr 9, 2018, at 1:45 PM, Vineet Garg 
> wrote:

Hello,

The branch for 3.0.0 release has been cut off (branch-3.0.0).

I am going to update all unresolved JIRA’s which aren’t marked blocker with fix 
version 3.0.0 to 3.1.0. Please update them if you would like to get your patch 
in 3.0.0.

Thanks,
Vineet G

[jira] [Created] (HIVE-19137) orcfiledump doesn't print hive.acid.version value

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19137:
-

 Summary: orcfiledump doesn't print hive.acid.version value
 Key: HIVE-19137
 URL: https://issues.apache.org/jira/browse/HIVE-19137
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


HIVE-18659 added hive.acid.version in the file footer.  
orcfiledump prints something like 
{noformat}
User Metadata:
  hive.acid.key.index=1,536870912,1;
  hive.acid.stats=2,0,0
  hive.acid.version=
{noformat}

probably because
public static void setAcidVersionInDataFile(Writer writer) {
  //so that we know which version wrote the file
  ByteBuffer bf = ByteBuffer.allocate(4).putInt(ORC_ACID_VERSION);
  bf.rewind(); //don't ask - some ByteBuffer weridness. w/o this, empty 
buffer is written
  writer.addUserMetadata(ACID_VERSION_KEY, bf);
}
{noformat}

use 
{{UTF8.encode()) instead



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Apache Hive 3.0.0 Release branch cutoff

[Vineet Garg]

Hello,

The branch for 3.0.0 release has been cut off (branch-3.0.0).

I am going to update all unresolved JIRA’s which aren’t marked blocker with fix 
version 3.0.0 to 3.1.0. Please update them if you would like to get your patch 
in 3.0.0.

Thanks,
Vineet G

[jira] [Created] (HIVE-19136) DbNotifications clean up throws NPE on mysql databases

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-19136:
--

 Summary: DbNotifications clean up throws NPE on mysql databases
 Key: HIVE-19136
 URL: https://issues.apache.org/jira/browse/HIVE-19136
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


I see following stack trace in HMS logs when db notification cleaner thread 
tries to clean the old notification.

{noformat}
Exception in thread "CleanerThread" javax.jdo.JDODataStoreException: 
Transaction failed to commit
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
at 
org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:189)
at 
org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:790)
at 
org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:10425)
at 
org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:9258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy22.cleanNotificationEvents(Unknown Source)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener$CleanerThread.run(DbNotificationListener.java:737)
NestedThrowablesStackTrace:
Unexpected exception encountered during query.
org.datanucleus.exceptions.NucleusDataStoreException: Unexpected exception 
encountered during query.
at 
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.close(ConnectionFactoryImpl.java:569)
at 
org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionRolledBack(ConnectionManagerImpl.java:400)
at 
org.datanucleus.TransactionImpl.internalRollback(TransactionImpl.java:534)
at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:451)
at 
org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
at 
org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:790)
at 
org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:10425)
at 
org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:9258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy22.cleanNotificationEvents(Unknown Source)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener$CleanerThread.run(DbNotificationListener.java:737)
Caused by: java.sql.SQLException: Unexpected exception encountered during query.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:963)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2582)
at 
com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4698)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4590)
at 
com.zaxxer.hikari.pool.ProxyConnection.close(ProxyConnection.java:233)
at 
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.close(ConnectionFactoryImpl.java:557)
... 14 more
Caused by: java.lang.NullPointerException
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
... 18 more
Nested Throwables StackTrace:
java.sql.SQLException: Unexpected exception encountered during query.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:963)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2582)
at 
com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4698)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4590)
at 

[jira] [Created] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade

[Alan Gates (JIRA)]

Alan Gates created HIVE-19135:
-

 Summary: Need tool to allow admins to create catalogs and move 
existing dbs to catalog during upgrade
 Key: HIVE-19135
 URL: https://issues.apache.org/jira/browse/HIVE-19135
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates


As part of upgrading to Hive 3 admins may wish to create new catalogs and move 
some existing databases into those catalogs.  We can do this by adding options 
to schematool.  This guarantees that only admins can do these operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review200741
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 217 (original), 222 (patched)


If the catalogs.add(rawStore.getCatalog(catName)) is moved to another line, 
safer to use {} for a  block under for loop.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
Lines 118 (patched)


I think we can just keep track of size of partitions in it (and perhaps 
tables) since that should be taking almost all the memory.

I am just worried that SharedCache is more complicated structure, and we 
might hit bugs around such structures in estimation. There are whole lot of 
other basic classes that we don't estimate size of.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 1786 (patched)


In master the Validator impls have been moved to separate classes (recent 
commit I suppose). Can yo u also please add SizeValidator as a seperate class ?


- Thejas Nair


On April 9, 2018, 9:57 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> ---
> 
> (Updated April 9, 2018, 9:57 a.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-19126
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
>  6f4ec6f1ea 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
>  2f7fa24558 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
>  0bbaf7e459 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  c47856de87 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  89b400697b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  995137f967 
> 
> 
> Diff: https://reviews.apache.org/r/66503/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Review Request 66514: HIVE-17645 MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66514/
---

Review request for hive, Eugene Koifman and Sergey Shelukhin.


Repository: hive-git


Description
---

Replace usage of SessionState.getTxnMgr() from several places, by doing some 
refactoring to make the TxnManager available in fields passed in during 
construction/initialization:
- SemanticAnalyzer.genFileSinkPlan()
- ReplicationSemanticAnalyzer.analyzeReplLoad()
- LoadSemanticAnalyzer.analyzeExternal()
- ImportSemanticAnalyzer.prepareImport()
- DDLSemanticAnalyzer.handleTransactionalTable()


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java fb1efe01dc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java a8d851fd81 
  ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java 
6b333d7184 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadConstraint.java
 60c85f58e5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java
 bc7d0ad0b9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 06adc64727 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
 1395027159 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/Context.java
 bb51f36a25 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
9e66422904 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
8b639f7922 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
e49089b91e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
79b2e48ee2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ff0a2e6a1b 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/MessageHandler.java
 3ccd639d62 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/TableHandler.java 
4cd75d8128 


Diff: https://reviews.apache.org/r/66514/diff/1/


Testing
---


Thanks,

Jason Dere

[jira] [Created] (HIVE-19134) Update copyright NOTICE

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19134:
--

 Summary: Update copyright NOTICE
 Key: HIVE-19134
 URL: https://issues.apache.org/jira/browse/HIVE-19134
 Project: Hive
  Issue Type: Task
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0
 Attachments: HIVE-19134.1.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly

[Bharathkrishna Guruvayoor Murali (JIRA)]

Bharathkrishna Guruvayoor Murali created HIVE-19133:
---

 Summary: HS2 WebUI phase-wise performance metrics not showing 
correctly
 Key: HIVE-19133
 URL: https://issues.apache.org/jira/browse/HIVE-19133
 Project: Hive
  Issue Type: Bug
Reporter: Bharathkrishna Guruvayoor Murali
Assignee: Bharathkrishna Guruvayoor Murali
 Attachments: WebUI-compile time query metrics.png

The query specific WebUI metrics (go to drilldown -> performance logging) are 
not showing up in the correct phase and are often mixed up.
Attaching screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19132) Ctrl+Z ; fg breaks beeline history functionality

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-19132:
---

 Summary: Ctrl+Z ; fg breaks beeline history functionality
 Key: HIVE-19132
 URL: https://issues.apache.org/jira/browse/HIVE-19132
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Zoltan Haindrich


I tend to use the terminal's ability to suspend the running program; do 
something quick ; and return to ithowever beeline doesn't like it; and the 
history doesn't work anymore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19131) DecimalColumnStatsMergerTest comparison review

[Laszlo Bodor (JIRA)]

Laszlo Bodor created HIVE-19131:
---

 Summary: DecimalColumnStatsMergerTest comparison review
 Key: HIVE-19131
 URL: https://issues.apache.org/jira/browse/HIVE-19131
 Project: Hive
  Issue Type: Bug
Reporter: Laszlo Bodor


DecimalColumnStatsMergerTest has a strange comparison logic, which needs to be 
reviewed.

Regarding low and high values, it uses compareTo with the same direction, which 
seems to be incorrect.

{code}

Decimal lowValue = aggregateData.getLowValue() != null && 
(aggregateData.getLowValue().compareTo(newData.getLowValue()) > 0) ? 
aggregateData .getLowValue() : newData.getLowValue(); 
aggregateData.setLowValue(lowValue); Decimal highValue = 
aggregateData.getHighValue() != null && 
(aggregateData.getHighValue().compareTo(newData.getHighValue()) > 0) ? 
aggregateData .getHighValue() : newData.getHighValue();

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66369: HIVE-18839

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66369/
---

(Updated April 9, 2018, 12:15 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18839
https://issues.apache.org/jira/browse/HIVE-18839


Repository: hive-git


Description
---

HIVE-18839


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
0627c3537840757dae4061e4e729a48bee9d2ec9 
  itests/src/test/resources/testconfiguration.properties 
28c14ebc4cede903f19cca79996ccc441774d147 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
fb1efe01dcddb4fd77f05918e46d742922dd313d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
50fc4e0c63a060781c93f0e65c5da32dd9116d70 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
5a95649f5bccbdd44cb4c4d12c65d39a5a733a56 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java 
490f3b8ffe7564d10d86293b402d3c0dad2f7ee7 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
c8cafa2a68f897b1034e5bf61e872e044e01c22a 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
38fbb7ba53d6bae1c0ea8800a97c025ff8e1b8a6 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 53dc8ec1974dfc095bd5c7601c8d486b4712319e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAggregateIncrementalRewritingRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAugmentMaterializationRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveNoAggregateIncrementalRewritingRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/MaterializedViewRewritingRelVisitor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
41de17fd4679009ef6a4fb5a6d976cbc794ce791 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/MaterializedViewRebuildSemanticAnalyzer.java
 75eb50c5797b312f66e81b5cec23849684e641fc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
ff0a2e6a1beef08faca3d108c17330af26516934 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
4c86fb89376041b30bc5c90a4e84f2e685d8ff82 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_4.q 
c7f050be8da71f13463c22df2ccdab21c0e40b6a 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_5.q 
PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out
 1ef7b876d82136e3a7ad9021db8c293a3cc4808a 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_4.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_5.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_rebuild_dummy.q.out
 1ef7b876d82136e3a7ad9021db8c293a3cc4808a 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_7.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out 
18eb1d1daa483d4635db514beb96104b558ee1bd 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_2.q.out 
144f9d84208ab32528a20bc6151c9be58571291e 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 
65614566c9b738ea0354dc6d206a0ed7e635174e 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out 
48c0ecb23f1657c99ef30ed6963a3af1737f0514 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out
 32b408f5e4d389f9bd143695f8dd570db234a876 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_time_window.q.out
 bfbac3f3492fdd15b55475cd423cf262793cff84 
  ql/src/test/results/clientpositive/materialized_view_rewrite_1.q.out 
75e828fd4296d5f956916ba785ca664bfcb6 
  ql/src/test/results/clientpositive/materialized_view_rewrite_2.q.out 
6e8b2e3e6593b9e921517cb349755a7b668dc900 
  ql/src/test/results/clientpositive/materialized_view_rewrite_3.q.out 
e3bd233177a9058cb202e641175a5fbb4aa7ca89 
  ql/src/test/results/clientpositive/materialized_view_rewrite_4.q.out 
7301571cbf06b874fa027aa8e7a9bc1192a5c948 
  ql/src/test/results/clientpositive/materialized_view_rewrite_5.q.out 
bab3cacc2ce420578732037fb2cefc3d375c4544 
  ql/src/test/results/clientpositive/materialized_view_rewrite_6.q.out 
017d793283904ca83fc1074eb01ca9879805a61a 
  ql/src/test/results/clientpositive/materialized_view_rewrite_7.q.out 
486a50d55a346b7c2dd1e4e30a70ac7663da1d5d 
  ql/src/test/results/clientpositive/materialized_view_rewrite_8.q.out 
1ca06d3cc4207818f059650b0bc250433d41c783 
  ql/src/test/results/clientpositive/materialized_view_rewrite_9.q.out 
3120e0db7a19017d9edecc541f5bf1a1d444e66d 
  

Re: Review Request 66369: HIVE-18839

[Jesús Camacho Rodríguez]

> On April 7, 2018, 12:29 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1496 (patched)
> > 
> >
> > Lets move this class to  
> > org.apache.hadoop.hive.ql.optimizer.calcite.rules package. It will be 
> > easier to find it. Also, it doesn't belong in Hive class.

Moved to org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.


> On April 7, 2018, 12:29 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1531 (patched)
> > 
> >
> > It will be good to add comment on what filter we construct.
> > ROW_ID.transactionid <= high_watermark and ROW_ID.transactionid not in 
> > (invalid_txns)

Added comment in the rule.


> On April 7, 2018, 12:29 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAggregateIncrementalRewritingRule.java
> > Lines 119 (patched)
> > 
> >
> > Don't we handle count() too ?

We do. When we rollup the COUNT, we will find a SUM on top of the union. Hence, 
we only need to handle SUM here. Added a comment.


> On April 7, 2018, 12:29 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 970 (patched)
> > 
> >
> > It will be good to add why we need to do it this way which is CBO 
> > doesn't yet support merge.

Added comment to method.


> On April 7, 2018, 12:29 a.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsInvalidationCache.java
> > Lines 66-68 (original), 68-70 (patched)
> > 
> >
> > Need to update comments.

This info seems up-to-date. I have just added that there is also information 
about 'whether there was any update or delete in the source tables since the 
materialized view was created'. Is that what you meant or there is something 
else missing?


> On April 7, 2018, 12:29 a.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsRebuildLockHandler.java
> > Lines 115 (patched)
> > 
> >
> > ||  prevResourceLock.state.get() != State.COMMIT_READY

prevResourceLock.state.get() != State.ACQUIRED is correct. If the state is 
State.ACQUIRED, refresh is valid. But if the state is State.COMMIT_READY, there 
is no need to refresh anymore.


> On April 7, 2018, 12:29 a.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsRebuildLockHandler.java
> > Lines 135 (patched)
> > 
> >
> > || state != NOT_ACQUIRED

There is no need here to check for NOT_ACQUIRED. When a lock state is 
NOT_ACQUIRED, the state is not stored in the handler (you can see in L83 that 
we return the response immediately but we do not store it).


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66369/#review200681
---


On April 5, 2018, 2:23 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66369/
> ---
> 
> (Updated April 5, 2018, 2:23 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18839
> https://issues.apache.org/jira/browse/HIVE-18839
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18839
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 5f07312651dc64fe23dcb21d185d6676f33f3bb0 
>   itests/src/test/resources/testconfiguration.properties 
> d2e077b5096cbf82f7b8a98e8f66a5ef98e72bc9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> fb1efe01dcddb4fd77f05918e46d742922dd313d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
> 50fc4e0c63a060781c93f0e65c5da32dd9116d70 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
> 5a95649f5bccbdd44cb4c4d12c65d39a5a733a56 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java 
> 490f3b8ffe7564d10d86293b402d3c0dad2f7ee7 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> c8cafa2a68f897b1034e5bf61e872e044e01c22a 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 

Re: Review Request 66369: HIVE-18839

[Jesús Camacho Rodríguez]

> On April 3, 2018, 11:34 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 1460 (patched)
> > 
> >
> > This will update all MV before they are considered for rewrite. Looks 
> > like we are doing extra work which could be avoided by delaying augmenting 
> > MV until it qualifies for rewrite.
> 
> Jesús Camacho Rodríguez wrote:
> Rewriting algorithm needs those filters to produce correct rewriting, 
> hence though we are indeed doing extra work, I do not think we can avoid it. 
> This part is also useful when MVs are outdated (not only for rebuild), as 
> they may still be used to produce correct rewritings.
> 
> On the other hand, we could avoid this work by implementing the 
> pre-filtering techniques that we were talking about at some point. The idea 
> would be that only views relevant for a given query will be returned by the 
> method, then they would be enriched.
> 
> Another improvement is that for a rebuild operation, we only consider in 
> the rewriting the MV that we are rebuilding? However, this is a kind of 
> heuristic, since other MVs that are up-to-date may be useful to rewrite the 
> rebuild query.
> 
> Ashutosh Chauhan wrote:
> Yeah, this makes pre-filtering even more important. I thought we had 
> prefiltering based on table and col names. If not, we should add that in 
> follow-up.
> Heuristic for rebuidling makes lot of sense. Since I dont expect 
> overlapping MVs to be common. Heuristic will also improve compile time.

Prefiltering based on tables in present in Calcite, but we need to add the 
materializations to the planner first. Here logic is a bit different, because 
we are enriching the materializations before adding them to the planner. In any 
case, for the rebuild, which is the most common case of incremental rewriting, 
adding the heuristic will give us a boost. I just did it and added comments 
explaining it.

FWIW, in the creation metadata we have information about the tables used by a 
materialization, so we could do a simple prefilter before loading the 
materializations in the planner (follow-up).


> On April 3, 2018, 11:34 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAggregateIncrementalRewritingRule.java
> > Lines 57-60 (patched)
> > 
> >
> > Is this rewrite always better?
> > 
> > Imagine tabA and tabB to be in millions rows but their join generates 
> > cross product of billions of rows. In prev case we were doing union of 
> > large MV with join of two tables (with which filter may get very small) as 
> > compared to here where we do ROJ of MV with one source table.
> 
> Jesús Camacho Rodríguez wrote:
> Comment may be misleading. In the last MERGE, _source_ refers to _source_ 
> block, not to a single query:
> _(SELECT a, b, SUM(x) AS s, COUNT(*) AS c --NEW DATA
>  FROM TAB_A
>  JOIN TAB_B ON (TAB_A.a = TAB_B.z)
>  WHERE TAB_A.ROW_ID > 5
>  GROUP BY a, b) source_
> 
> In any case, I think the point about performance is relevant. I expect if 
> we do a rebuild with not many new rows, incremental maintenance will get the 
> edge. However, if there are many changes in source tables, union may be 
> benefitial over ROJ? But in the latter case, we also need to take into 
> account that the UNION operation will invalidate the LLAP cache of the MV, 
> while the ROJ variant will not. I can add a flag to control the incremental 
> maintenance piece, even if it is always true, in case we want to disable it 
> to do some tests in the future? What do you think?
> 
> Ashutosh Chauhan wrote:
> yeah.. lets add a flag with default to true.

Added hive.materializedview.rebuild.incremental.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66369/#review200335
---


On April 5, 2018, 2:23 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66369/
> ---
> 
> (Updated April 5, 2018, 2:23 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18839
> https://issues.apache.org/jira/browse/HIVE-18839
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18839
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 5f07312651dc64fe23dcb21d185d6676f33f3bb0 
>   itests/src/test/resources/testconfiguration.properties 
> d2e077b5096cbf82f7b8a98e8f66a5ef98e72bc9 
>   

Re: Apache Hive 3.0.0 release preparation

[Zoltan Haindrich]

Hello

A few weeks earlier I've tried to hunt down this problem...
so...to my best knowledge the cause of this seems to be the following:

* in some cases the "cleanup" after a failed query may somehow leave some 
threads behind...
* these threads have reference to the "customized" session classloader - this 
makes the threads more memory hungry
* after a while these threads/classloaders eat up the heap...

I've opened HIVE-18522 for this thread issue

I think this problem is not new ...and it might have been present earlier as 
well...the only thing what changed is that there were a few more new features 
which have added new udfs/etc which made the memory cost of a session more 
heavier..
...and as a sidenote: I'm not convinced that this issue will arise in a proper 
hs2 setup - as it might be easily connected to the fact that these tests are 
using the cli driver to execute the tests.


cheers,
Zoltan

On 7 Apr 2018 7:15 p.m., Ashutosh Chauhan  wrote:
We need to investigate and find out root cause of these failures. If its
determined that its a corner case and fix is non-trivial then we may
release note it under known issues. But ideally we should fix these
failures.
Cutting a branch should make it easier since branch is expected to receive
lot less commits as compared to master so it should be faster to stabilize
branch.

On Fri, Apr 6, 2018 at 10:49 AM, Eugene Koifman 
wrote:

> Cutting the branch before the tests are stabilized would mean we have to
> fix them in 2 places.
>
> On 4/6/18, 10:05 AM, "Thejas Nair"  wrote:
>
> That needs to be cleaned up. There are far too many right now, its
> just not handful of flaky tests.
>
>
> On Fri, Apr 6, 2018 at 2:48 AM, Peter Vary  wrote:
> > Hi Team,
> >
> > I am new to the Hive release process and it is not clear to me how
> the failing tests are handled. Do we plan to fix the failing tests before
> release? Or it is accepted to cut a new major release with known test
> issues.
> >
> > Thanks,
> > Peter
> >
> >> On Apr 5, 2018, at 8:25 PM, Vineet Garg 
> wrote:
> >>
> >> Hello,
> >>
> >> I plan to cut off branch for Hive 3.0.0 on Monday (9 April) since
> bunch of folks have big patches pending.
> >>
> >> Regards,
> >> Vineet G
> >>
> >>> On Apr 2, 2018, at 3:14 PM, Vineet Garg 
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> We have enough votes to prepare a release candidate for Hive
> 3.0.0. I am going to cutoff a branch in a day or two. I’ll send an email as
> soon as I have the branch ready.
> >>> Meanwhile there are approximately 69 JIRAs which are currently
> opened with fix version 3.0.0. I’ll appreciate if their respective owners
> would update the JIRA if it is a blocker. Otherwise I’ll update them to
> defer the fix version to next release.
> >>>
> >>> Regards,
> >>> Vineet G
> >>>
> >>
> >
>
>
>
>

[jira] [Created] (HIVE-19130) NPE thrown when applied drop partition event on target.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-19130:
---

 Summary: NPE thrown when applied drop partition event on target.
 Key: HIVE-19130
 URL: https://issues.apache.org/jira/browse/HIVE-19130
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, repl
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


During incremental replication, if we split the events batch as follows, then 
the REPL LOAD on second batch throws NPE.

Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1)

Batch-2: DROP_TABLE(t1) ->  CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> 
DROP_PARTITION (t1.p1)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
---

Review request for hive and Thejas Nair.


Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-19126


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 6f4ec6f1ea 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 2f7fa24558 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 0bbaf7e459 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856de87 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 89b400697b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 995137f967 


Diff: https://reviews.apache.org/r/66503/diff/1/


Testing
---


Thanks,

Vaibhav Gumashta