HIVE-18373 Review Request

[Madhudeep petwal]

Hi,

I am working on this  JIRA
 .
My patch have received +1 from Zoltan Haindrich

Please guide me for further steps.

Thanks
Madhudeep Petwal

Re: Review Request 65422: HIVE-17626

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review197649
---




ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
Lines 131 (patched)


This is hackish.. as pointed above it needs to happen via explain modifier.



ql/src/java/org/apache/hadoop/hive/ql/Context.java
Lines 72 (patched)


Use  java.util.Optional instead



ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
Lines 21 (patched)


Use java's nonnull annotation.



ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java
Lines 40 (patched)


Need to check for 
org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError and 
java.lang.OutOfMemoryError only.



ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 43 (patched)


Need to check for 
org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError and 
java.lang.OutOfMemoryError only.



ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 54 (patched)


Why is this needed?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 139 (patched)


Should also compare with parent to check rowcount and other stats are not 
greater than parent.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 310 (patched)


Should also compare with parent to check rowcount and other stats are not 
greater than parent and normalize to parent stats in that case.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 2314 (patched)


Should also compare with parent to check rowcount and other stats are not 
greater than parent and normalize to parent stats in that case.



ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java
Lines 36 (patched)


A flat map of operators looses hierarichal info in which operators are 
organized which is tree. So, this match needs to happen  via sub-graph matching 
pattern. See SharedWorkOptimizer::areMergeable() .



ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java
Lines 75 (patched)


LOG.debug



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java
Lines 64 (patched)


throw MapJoinMemoryExhaustionError


- Ashutosh Chauhan


On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated Jan. 30, 2018, 6:13 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java 
> ad31287879 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 
> 533f0bcd6f 
>   itests/src/test/resources/testconfiguration.properties d86ff58840 
>   ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08 
>   ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  8dd7cfe58c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
>  134fc0ff0b 
>   
> 

[GitHub] hive pull request #308: HIVE-17990 Add Thrift and DB storage for Schema Regi...

[alanfgates]

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/308

HIVE-17990 Add Thrift and DB storage for Schema Registry objects



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17990

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/308.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #308


commit a5a49e9a0fcca40c298849e5b9d2eada00cebd9a
Author: Alan Gates 
Date:   2017-10-19T23:49:38Z

HIVE-17990 Add Thrift and DB storage for Schema Registry objects




---

[jira] [Created] (HIVE-18727) Update GenericUDFEnforceNotNullConstraint to throw an ERROR instead of Exception on failure

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-18727:
--

 Summary: Update GenericUDFEnforceNotNullConstraint to throw an 
ERROR instead of Exception on failure
 Key: HIVE-18727
 URL: https://issues.apache.org/jira/browse/HIVE-18727
 Project: Hive
  Issue Type: Improvement
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0


Throwing an exception makes TezProcessor stop retrying the task. Since this is 
NOT NULL constraint violation we don't want TezProcessor to keep retrying on 
failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18726) Implement DEFAULT constraint

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-18726:
--

 Summary: Implement DEFAULT constraint
 Key: HIVE-18726
 URL: https://issues.apache.org/jira/browse/HIVE-18726
 Project: Hive
  Issue Type: New Feature
  Components: Query Planning, Query Processor
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18725) Improve error handling for subqueries if there is wrong column reference

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-18725:
--

 Summary: Improve error handling for subqueries if there is wrong 
column reference
 Key: HIVE-18725
 URL: https://issues.apache.org/jira/browse/HIVE-18725
 Project: Hive
  Issue Type: Improvement
  Components: Query Planning
Reporter: Vineet Garg


If there is a column reference within subquery which doesn't exist Hive throws 
misleading error message.
e.g. 

{code:sql}
select * from table1 where table1.col1 IN (select col2 from table2 where 
table2.col1=table1.non_existing_column) and table1.col1 IN (select 4);
{code}

The above query, assuming table1 doesn't have non_existing_column, will throw 
following misleading error:
{noformat}
FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'col1': 
Only 1 SubQuery expression is supported.
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18724) Improve error handling for subqueries referencing columns(correlated) of it's grand-parent query

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-18724:
--

 Summary: Improve error handling for subqueries referencing 
columns(correlated) of it's grand-parent query
 Key: HIVE-18724
 URL: https://issues.apache.org/jira/browse/HIVE-18724
 Project: Hive
  Issue Type: Improvement
  Components: Query Planning
Reporter: Vineet Garg


Currently Hive doesn't allow correlated subqueries which refer to columns of 
grand parent query e.g.

{code:sql}
select t1.a from t t1 where t1.b IN (select t2.c from t t2 where t2.a IN 
(select max(t3.b) from t t3, t t4 where t3.a=t2.a and t3.b=t1.b))
 {code}

In this example inner most query has column reference {{t3.b=t1.b}} where 
{{t1.b}} references outer most query.

The error which Hive compiler throw is 
{noformat}
FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'a': Nested 
SubQuery expressions are not supported.
{format}

This is very misleading since Hive does support Nested subqueries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18723) CompactorOutputCommitter.commitJob() - check rename() ret val

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-18723:
-

 Summary: CompactorOutputCommitter.commitJob() - check rename() ret 
val
 Key: HIVE-18723
 URL: https://issues.apache.org/jira/browse/HIVE-18723
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman



right now ret val is ignored {{fs.rename(fileStatus.getPath(), newPath); }}

Should this use {{FileUtils.ename(FileSystem fs, Path sourcePath, Path 
destPath, Configuration conf) }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18722) Vectorization: Adding SUM(HASH(..)) to full query seems to produce flakey results -- need to investiage

[Matt McCline (JIRA)]

Matt McCline created HIVE-18722:
---

 Summary: Vectorization: Adding SUM(HASH(..)) to full query seems 
to produce flakey results -- need to investiage
 Key: HIVE-18722
 URL: https://issues.apache.org/jira/browse/HIVE-18722
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline


When added to HIVE-18622 changes, the query results vary from laptop results 
when run on Hive QA cluster.  Need to investigate after HIVE-18622 commits.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65663: HIVE-18710 extend inheritPerms to ACID in Hive 2.X

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65663/#review197625
---




ql/src/java/org/apache/hadoop/hive/ql/Context.java
Lines 366 (patched)


This is a staging dir... there is no need to inherit perms for temp dir.



ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
Line 210 (original), 211 (patched)


whitespace.



ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
Line 225 (original), 227 (patched)


inherit perms feature till now is for insert statements, not for 
concat/merge files. If you want to increase the scope of a feature we shall do 
it in a seperate jira and limit this jira for acid tables perm inheritance.



ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
Line 264 (original), 265 (patched)


inherit perms feature till now is for insert statements, not for 
concat/merge files. If you want to increase the scope of a feature we shall do 
it in a seperate jira and limit this jira for acid tables perm inheritance.



ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java
Lines 75 (patched)


inherit perms feature till now is for insert statements, not for load/copy 
files. If you want to increase the scope of a feature we shall do it in a 
seperate jira and limit this jira for acid tables perm inheritance.



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Line 231 (original), 235 (patched)


Permission inheritance is taken care in MoveTask. Why is there a need to do 
this here?



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Line 240 (original), 245 (patched)


Permission inheritance is taken care in MoveTask. Why is there a need to do 
this here?



ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java
Line 109 (original), 112 (patched)


inherit perms feature till now is for insert statements, not for 
replication. If you want to increase the scope of a feature we shall do it in a 
seperate jira and limit this jira for acid tables perm inheritance.



ql/src/java/org/apache/hadoop/hive/ql/exec/SkewJoinHandler.java
Line 323 (original), 324 (patched)


these are temp dir, no need for inheritance here.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2928 (patched)


Please read comment above, this is not needed.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 3181 (patched)


Because feature is for both perms and grp.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Line 3182 (original), 3186 (patched)


These are files. No need for recursive here.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 3214 (patched)


perms are already inherited. No need to call again.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Line 3268 (original), 3278 (patched)


increasing scope of feature.


- Ashutosh Chauhan


On Feb. 15, 2018, 1:28 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65663/
> ---
> 
> (Updated Feb. 15, 2018, 1:28 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 
> 2b7a57bb369f0fc3204157a15dff761c1e73419d 
>   common/src/java/org/apache/hadoop/hive/io/HdfsUtils.java 
> 16fc96eae99705a0625831122afcef3586fe490e 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 
> 2ed1c6e00417f713ba8ded16b7591973b7e64271 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 6bba057e764099ff2e6ef974577cb6b42fbecab0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 
> 2683f294f61f5e6e1c553c0095fcc11461229c04 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
> a9d03d060adeaa5cad6bef48a63c048f23819d01 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 
> 

Re: Unit Test Failing

[Madhudeep petwal]

Hi,

My patch have received +1 from Zoltan Haindrich

Please guide me for further steps.

Thanks

On Wed, Feb 7, 2018 at 12:28 PM, Vihang Karajgaonkar 
wrote:

> I took a quick look at the failing tests on your patch and they all look to
> be the regularly failing ones. Another way to confirm if this is true is by
> going to the Precommit job's Test Result link posted on your JIRA and
> checking if Jenkins shows that the test has been failing x number builds.
> Most committers know the usual tests which have been failing for a while.
> Of-course, you can make their job easier too by verifying if the suspicious
> tests also fail without your patch so that you can rule out your changes as
> the ones causing failure ;) Hope that helps.
>
> On Tue, Feb 6, 2018 at 9:09 PM, Madhudeep petwal <
> madhudeep11pet...@gmail.com> wrote:
>
> > Hi
> >
> > I am a new committer to Hive.
> > I am working on this JIRA  > a/browse/HIVE-18373>
> >  .
> >
> > I have already submitted the patch and various unit test cases are
> failing
> > which are not related to my code.
> >
> > There is a mail by Vihang Karajgaonkar by subject as *[DISCUSS] 26 unit
> > test failing regularly*.
> > I had tested my code before and it was working, but after some commits,
> > there are failing of test cases.
> >
> > Please someone help me out.
> >
> > Thanks
> > Madhudeep Petwal
> >
>

Review Request 65668: HIVE-18721

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65668/
---

Review request for hive, Gopal V and Jason Dere.


Bugs: HIVE-18721
https://issues.apache.org/jira/browse/HIVE-18721


Repository: hive-git


Description
---

Bucket Map Join : Handle empty buckets

Bucket Map Join needs to bluff Tez by sending empty task list for 
DataMovementEvent for those buckets for which there is no data.


Diffs
-

  itests/src/test/resources/testconfiguration.properties c2252f37c1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionEdge.java 
1ac1d14484 
  ql/src/test/queries/clientpositive/bucket_map_join_tez_empty.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez_empty.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65668/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-18721) Bucket Map Join : Handle empty buckets

[Deepak Jaiswal (JIRA)]

Deepak Jaiswal created HIVE-18721:
-

 Summary: Bucket Map Join : Handle empty buckets
 Key: HIVE-18721
 URL: https://issues.apache.org/jira/browse/HIVE-18721
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


Bucket Map Join needs to bluff Tez by sending empty task list for 
DataMovementEvent for those buckets for which there is no data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #307: HIVE-18720 : Replicate Commit Txn operation (without...

[maheshk114]

GitHub user maheshk114 opened a pull request:

https://github.com/apache/hive/pull/307

HIVE-18720 : Replicate Commit Txn operation (without writes)

Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions 
modified within the txn.

Source warehouse:

Create EVENT_COMMIT_TXN event type with corresponding message format etc.
Target warehouse:

Repl load should read this event from the dump.
Validate the source txn ID from the event using the Source-Target Txn ID 
map maintained in target metastore. Also, need to check if corresponding target 
txn ID is valid.
If valid, then apply the event and commit the corresponding target 
transaction.
This new event should be idempotent such that if it is applied twice, then 
second time it should be loop.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maheshk114/hive BUG_96430

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/307.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #307


commit 4a855de860ccec6e37e4c16ebbddee575e9ae2f2
Author: Mahesh Kumar Behera 
Date:   2018-02-12T05:07:01Z

HIVE-18679 : create/replicate open transaction event

commit c745a4066b31075004b96200da079b4dd4fd2743
Author: Mahesh Kumar Behera 
Date:   2018-02-12T14:29:54Z

HIVE-18679 : create/replicate open transaction event : rebased with Alan's 
change

commit 4f7c15617ff0a84b6373d5fc0405e53a9c89703b
Author: Mahesh Kumar Behera 
Date:   2018-02-14T15:56:51Z

HIVE-18720 : Replicate Commit Txn operation (without writes)




---