Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

[Harish Jaiprakash]

> On Oct. 27, 2017, 11:27 a.m., Harish Jaiprakash wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
> > Lines 878 (patched)
> > 
> >
> > Desc is overwritten in the loop, so only the last alter statement will 
> > take into effect.

Sorry about this. I went through the code, it works fine.


- Harish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/#review189399
---


On Oct. 27, 2017, 6:18 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63346/
> ---
> 
> (Updated Oct. 27, 2017, 6:18 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 16190dc0a1 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 4c35380af4 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql ca7af06e7a 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 0ee0f758f1 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 86f3e5c4a9 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 1affa7a119 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 2453bb92c6 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 0ee0f758f1 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 47bfc5ae08 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql aa27a6486b 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 5c770e2659 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 87f6e1d64e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  f1258bab17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  9df1b82d62 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> dc29b122c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 99ed71a5fa 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 053393ce51 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> f0658a665e 
>   ql/src/test/queries/clientpositive/resourceplan.q 586491dc91 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 2231bb566d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  3deef0ee96 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  3ff924fe00 
>   standalone-metastore/src/main/resources/package.jdo 296e273ec8 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 1d455463a6 
> 
> 
> Diff: https://reviews.apache.org/r/63346/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

[Harish Jaiprakash]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/#review189399
---




ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
Lines 878 (patched)


Desc is overwritten in the loop, so only the last alter statement will take 
into effect.



ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java
Lines 63 (patched)


Should this be called defaultPoolPath, we should only use pool path 
everywhere right?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 9608 (patched)


We should filter this by rp_id too.



standalone-metastore/src/main/thrift/hive_metastore.thrift
Line 1050 (original), 1051 (patched)


I made a mistake here, it should be called poolPath, not pool name. We do 
not need parentPoolName and poolName when we have poolPath.


- Harish Jaiprakash


On Oct. 27, 2017, 6:18 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63346/
> ---
> 
> (Updated Oct. 27, 2017, 6:18 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 16190dc0a1 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 4c35380af4 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql ca7af06e7a 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 0ee0f758f1 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 86f3e5c4a9 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 1affa7a119 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 2453bb92c6 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 0ee0f758f1 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 47bfc5ae08 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql aa27a6486b 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 5c770e2659 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 87f6e1d64e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  f1258bab17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  9df1b82d62 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> dc29b122c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 99ed71a5fa 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 053393ce51 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> f0658a665e 
>   ql/src/test/queries/clientpositive/resourceplan.q 586491dc91 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 2231bb566d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  3deef0ee96 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  3ff924fe00 
>   standalone-metastore/src/main/resources/package.jdo 296e273ec8 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 1d455463a6 
> 
> 
> Diff: https://reviews.apache.org/r/63346/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63230: HIVE-17841 implement applying the resource plan

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63230/
---

(Updated Oct. 27, 2017, 2:43 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 892ebfa166 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AmPluginNode.java 35d380c92b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
d978a25b14 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapPluginEndpointClientImpl.java
 45c3e38dcc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
a326db3ab0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SessionExpirationTracker.java 
da93a3a791 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java b67c933b19 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
9b4714f1d7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
613522357e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
144816862d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
81d6b859a6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java 00501eef93 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
d725e90475 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 209cf57a6a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
59efd43be6 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
258a865fef 


Diff: https://reviews.apache.org/r/63230/diff/3/

Changes: https://reviews.apache.org/r/63230/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63230: HIVE-17841 implement applying the resource plan

[Sergey Shelukhin]

> On Oct. 27, 2017, 12:25 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java
> > Lines 120 (patched)
> > 
> >
> > Alternatively, since you are not looking at return value anyway. You 
> > can use CompletableFuture.allOf() which will returns one combined 
> > CompletableFuture which can be used to block.

FutureTask is not a completablefuture


> On Oct. 27, 2017, 12:25 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java
> > Lines 83 (patched)
> > 
> >
> > This could be doable without Guava. With JDK8 
> > CompletableFuture.acceptEither.

ScheduledFuture is not a completablefuture either... also it requires 
completionstage on top of that, which just by the length of the javadoc 
introduction seems to be designed for much more complex stuff :)


> On Oct. 27, 2017, 12:25 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Line 234 (original), 292 (patched)
> > 
> >
> > same here. awaitUninterruptibly()

Same reason :)


> On Oct. 27, 2017, 12:25 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Lines 491 (patched)
> > 
> >
> > When will init be cancelled?
> > 
> > If admin makes changes to pool, the event gets propagated and start 
> > with init state. Now if admin makes another change immediately (before the 
> > previous one completes), would that cancel the previous init?
> > 
> > Is applying EventState synchronous?

The only way to cancel init is from this thread; see pool.update and 
pool.destroy. So, if the pool is removed, pool.destroy would always be called 
in the same iteration with setting init state to cancelled.


> On Oct. 27, 2017, 12:25 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Lines 503 (patched)
> > 
> >
> > Assuming this happens when there is no capacity in cluster for spinning 
> > up new sessios (or could be other reasons)? Is there a way to determine why 
> > this failed and throw error if not restartable (may be some permission 
> > issue)?

It actually says internal error ;) This means the pool has been removed without 
init being marked as cancelled. Made the statement more specific


> On Oct. 27, 2017, 12:25 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Lines 1138 (patched)
> > 
> >
> > Create follow jira to handle this case? Killing all seems like overkill 
> > :P
> > 
> > We could kill most recently submitted queries that will satisfy delta. 
> > Or make it optional to the user to pick the policy they want (pluggable).

Will file a JIRA; this was a decision when we were discussing w/Carter and 
Gunther... because killing only some is "too complicated" :)


> On Oct. 27, 2017, 12:25 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Lines 1262 (patched)
> > 
> >
> > is this a recoverable state. If not should session be set to null here?

no, not recoverable... also if state is invalid master thread cannot handle it. 
It probably means someone added an enum constant.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63230/#review189373
---


On Oct. 24, 2017, 11:58 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63230/
> ---
> 
> (Updated Oct. 24, 2017, 11:58 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0b4abb824f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AmPluginNode.java 35d380c92b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
> d978a25b14 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapPluginEndpointClientImpl.java
>  45c3e38dcc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
> a326db3ab0 
>   
> 

[jira] [Created] (HIVE-17919) phase 2: don't kill all queries when a pool is resized down

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-17919:
---

 Summary: phase 2: don't kill all queries when a pool is resized 
down
 Key: HIVE-17919
 URL: https://issues.apache.org/jira/browse/HIVE-17919
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


It's actually easy to do but during some design discussions was deemed to be 
more confusing. Came up during code review.
When a pool is resized from e.g. 5 to 3 queries, and 4 are running, we only 
need to kill one query, not all 4.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17918) NPE during semijoin reduction optimization when LLAP caching disabled

[Jason Dere (JIRA)]

Jason Dere created HIVE-17918:
-

 Summary: NPE during semijoin reduction optimization when LLAP 
caching disabled
 Key: HIVE-17918
 URL: https://issues.apache.org/jira/browse/HIVE-17918
 Project: Hive
  Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere


DynamicValue (used by semijoin reduction optimization) relies on the 
ObjectCache. If LLAP cache is disabled then the DynamicValue is broken in LLAP:

{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:254)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
while processing row
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:928)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
... 18 more
Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value 
for RS_25_household_demographics_hd_demo_sk_min
at 
org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:130)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterLongColumnBetweenDynamicValue.evaluate(FilterLongColumnBetweenDynamicValue.java:80)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:39)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:41)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:112)
at 
org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:959)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:907)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:137)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:828)
... 19 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:61)
at 
org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:50)
at 
org.apache.hadoop.hive.ql.exec.ObjectCacheWrapper.retrieve(ObjectCacheWrapper.java:40)
at 
org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:123)
... 27 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:59)
... 30 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17917) VectorizedOrcAcidRowBatchReader.computeOffsetAndBucket optimization

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-17917:
-

 Summary: VectorizedOrcAcidRowBatchReader.computeOffsetAndBucket 
optimization
 Key: HIVE-17917
 URL: https://issues.apache.org/jira/browse/HIVE-17917
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Priority: Minor


VectorizedOrcAcidRowBatchReader.computeOffsetAndBucket optimization() 
computation is currently (after HIVE-17458) is done once per split.  It could 
instead be done once per file (since the result is the same for each split of 
the same file) and passed along in OrcSplit




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/#review189384
---




metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql
Lines 954 (patched)


Is this DEFAULT_POOL_NAME or default RP name (IIUC there is only one active 
RP and hence no need for default)? If later why do we need a default RP name?



metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql
Lines 7 (patched)


is this NON NULL by default?



metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql
Line 66 (original), 67 (patched)


if this is NULL, will it use DEFAULT_POOL_ID?


- Prasanth_J


On Oct. 27, 2017, 12:48 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63346/
> ---
> 
> (Updated Oct. 27, 2017, 12:48 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 16190dc0a1 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 4c35380af4 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql ca7af06e7a 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 0ee0f758f1 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 86f3e5c4a9 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 1affa7a119 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 2453bb92c6 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 0ee0f758f1 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 47bfc5ae08 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql aa27a6486b 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 5c770e2659 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 87f6e1d64e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  f1258bab17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  9df1b82d62 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> dc29b122c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 99ed71a5fa 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 053393ce51 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> f0658a665e 
>   ql/src/test/queries/clientpositive/resourceplan.q 586491dc91 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 2231bb566d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  3deef0ee96 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  3ff924fe00 
>   standalone-metastore/src/main/resources/package.jdo 296e273ec8 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 1d455463a6 
> 
> 
> Diff: https://reviews.apache.org/r/63346/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63343: HIVE-17766

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63343/
---

(Updated Oct. 27, 2017, 1:01 a.m.)


Review request for hive, Ashutosh Chauhan and Vineet Garg.


Bugs: HIVE-17766
https://issues.apache.org/jira/browse/HIVE-17766


Repository: hive-git


Description
---

HIVE-17766


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
3573d07e9a3bab4dabdc0b73292bd02ad05cbc89 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
46493ac006d00f2e6c0054da2438c0b01c1e3fd3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
5ccb69ac4824a2002b2bddb1ad206cb7acd3ace3 
  ql/src/test/queries/clientpositive/semijoin6.q PRE-CREATION 
  ql/src/test/results/clientpositive/semijoin2.q.out 
2c3c5d8db3d5422f21fde020199e878b87b0507f 
  ql/src/test/results/clientpositive/semijoin6.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/63343/diff/3/

Changes: https://reviews.apache.org/r/63343/diff/2-3/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 63343: HIVE-17766

[Jesús Camacho Rodríguez]

> On Oct. 26, 2017, 11:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 2136-2137 (original), 2119-2120 (patched)
> > 
> >
> > Shall we try to get equi joins keys and populate them in operator here? 
> > Will it happen later in planning.
> > Either way good to add a comment about this.

This is in Calcite operators, not in Hive. The reason keys are there is because 
semijoin extends equijoin. I do not know if they are effectively used, the 
logic here was pushing the computation of expressions to children project, but 
that seems to be happening when we trigger rewriting rules in any case, thus I 
do not think we need to do it here (maybe this code was written before some of 
the rules that take care of this were enabled in Hive?).


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63343/#review189370
---


On Oct. 26, 2017, 9:26 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63343/
> ---
> 
> (Updated Oct. 26, 2017, 9:26 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-17766
> https://issues.apache.org/jira/browse/HIVE-17766
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17766
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
> 3573d07e9a3bab4dabdc0b73292bd02ad05cbc89 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 46493ac006d00f2e6c0054da2438c0b01c1e3fd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5ccb69ac4824a2002b2bddb1ad206cb7acd3ace3 
>   ql/src/test/queries/clientpositive/semijoin6.q PRE-CREATION 
>   ql/src/test/results/clientpositive/semijoin2.q.out 
> 2c3c5d8db3d5422f21fde020199e878b87b0507f 
>   ql/src/test/results/clientpositive/semijoin6.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63343/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/
---

Review request for hive, Harish Jaiprakash and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 16190dc0a1 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 4c35380af4 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql ca7af06e7a 
  metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 0ee0f758f1 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 86f3e5c4a9 
  metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 1affa7a119 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 2453bb92c6 
  metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 0ee0f758f1 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 47bfc5ae08 
  metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql aa27a6486b 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 5c770e2659 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 87f6e1d64e 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
 f1258bab17 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
 9df1b82d62 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
dc29b122c9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 99ed71a5fa 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 053393ce51 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
f0658a665e 
  ql/src/test/queries/clientpositive/resourceplan.q 586491dc91 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 2231bb566d 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 3deef0ee96 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 3ff924fe00 
  standalone-metastore/src/main/resources/package.jdo 296e273ec8 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 1d455463a6 


Diff: https://reviews.apache.org/r/63346/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63230: HIVE-17841 implement applying the resource plan

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63230/#review189373
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java
Lines 120 (patched)


Alternatively, since you are not looking at return value anyway. You can 
use CompletableFuture.allOf() which will returns one combined CompletableFuture 
which can be used to block.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java
Lines 83 (patched)


This could be doable without Guava. With JDK8 
CompletableFuture.acceptEither.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 264 (patched)


I don't see value being used anywhere. Use Guava's Set.newIdentityHashSet() 
instead?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Line 234 (original), 292 (patched)


same here. awaitUninterruptibly()



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 332 (patched)


cosmetic nit: 
() -> {} lambda for runnable



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 363 (patched)


nit: improve comment to say reclaim/rebalance what?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 491 (patched)


When will init be cancelled?

If admin makes changes to pool, the event gets propagated and start with 
init state. Now if admin makes another change immediately (before the previous 
one completes), would that cancel the previous init?

Is applying EventState synchronous?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 503 (patched)


Assuming this happens when there is no capacity in cluster for spinning up 
new sessios (or could be other reasons)? Is there a way to determine why this 
failed and throw error if not restartable (may be some permission issue)?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 1138 (patched)


Create follow jira to handle this case? Killing all seems like overkill :P

We could kill most recently submitted queries that will satisfy delta. Or 
make it optional to the user to pick the policy they want (pluggable).



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 1223 (patched)


nit: all CAPS for enum



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 1262 (patched)


is this a recoverable state. If not should session be set to null here?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 1270 (patched)


Why handle unlock conditionally? handleSession* methods seems to be called 
only from here, so might as well unconditionally unlock in finally.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 1289 (patched)


Both American and British English in one line. Cancel[l]ed :)



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 1312 (patched)


these two lines seems to be repeating? may be encapsulate inside a 
session.clear() or session.reset() method.


- Prasanth_J


On Oct. 24, 2017, 11:58 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63230/
> ---
> 
> (Updated Oct. 24, 2017, 11:58 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0b4abb824f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AmPluginNode.java 35d380c92b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
> d978a25b14 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapPluginEndpointClientImpl.java
>  45c3e38dcc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
> a326db3ab0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SessionExpirationTracker.java 
> da93a3a791 
> 

[jira] [Created] (HIVE-17916) remove ConfVars.HIVE_VECTORIZATION_ROW_IDENTIFIER_ENABLED

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-17916:
-

 Summary: remove ConfVars.HIVE_VECTORIZATION_ROW_IDENTIFIER_ENABLED
 Key: HIVE-17916
 URL: https://issues.apache.org/jira/browse/HIVE-17916
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Teddy Choi


follow up from HIVE-12631.  Filing so it doesn't get lost.

There is this code in UpdateDeleteSemanticAnalyzer
{noformat}
  // TODO: remove when this is enabled everywhere
  HiveConf.setBoolVar(conf, 
ConfVars.HIVE_VECTORIZATION_ROW_IDENTIFIER_ENABLED, true);
{noformat}
The 1st update/delete statement on a session will enable this and it will be 
enabled for all future queries which makes this flag useless/misleading.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17915) Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over original acid files

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-17915:
-

 Summary: Enable VectorizedOrcAcidRowBatchReader to be used with 
LLAP IO elevator over original acid files
 Key: HIVE-17915
 URL: https://issues.apache.org/jira/browse/HIVE-17915
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Priority: Minor


Reading "original" acid files requires using 
_org.apache.hadoop.hive.ql.io.orc.RecordReader.getRowNumber()_ in 
__VectorizedOrcAcidRowBatchReader_
This API is not available on the reader that _LlapRecordReader_ provides so
_VectorizedOrcAcidRowBatchReader.canUseLlapForAcid()_ is used to disable LLAP 
IO in some corner cases.

It would be better if getRowNumber() was available for performance as well as 
simpler logic in the code.

This needs HIVE-17458 to be committed to make sense.

cc [~sershe], [~teddy.choi]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63343: HIVE-17766

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63343/#review189370
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 2136-2137 (original), 2119-2120 (patched)


Shall we try to get equi joins keys and populate them in operator here? 
Will it happen later in planning.
Either way good to add a comment about this.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Line 8257 (original), 8268 (patched)


comment: Adding a select operator to top of join to ensure projection of 
only left side columns.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 10 (patched)


No need for this test.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 14 (patched)


Please add explain for this query.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 16 (patched)


Please add explain for this query.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 18 (patched)


Please add explain for this query.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 20 (patched)


Please add explain for this query.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 22 (patched)


No need for this test.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 25 (patched)


One more test (with explain):
select * from tx1 u join tx2 v on (u.a + v.b > 400) and ((case when u.a > 3 
then true when v.b > 1900 then true else false) or (coalesce(u.a) + 
coalesce(v.b) > 1900) and u.a = v.a



ql/src/test/queries/clientpositive/semijoin6.q
Lines 26 (patched)


No need for this test.



ql/src/test/queries/clientpositive/semijoin6.q
Lines 28 (patched)


No need for this test.


- Ashutosh Chauhan


On Oct. 26, 2017, 9:26 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63343/
> ---
> 
> (Updated Oct. 26, 2017, 9:26 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-17766
> https://issues.apache.org/jira/browse/HIVE-17766
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17766
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
> 3573d07e9a3bab4dabdc0b73292bd02ad05cbc89 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 46493ac006d00f2e6c0054da2438c0b01c1e3fd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5ccb69ac4824a2002b2bddb1ad206cb7acd3ace3 
>   ql/src/test/queries/clientpositive/semijoin6.q PRE-CREATION 
>   ql/src/test/results/clientpositive/semijoin2.q.out 
> 2c3c5d8db3d5422f21fde020199e878b87b0507f 
>   ql/src/test/results/clientpositive/semijoin6.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63343/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63343: HIVE-17766

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63343/#review189356
---



Looks good to me

- Vineet Garg


On Oct. 26, 2017, 9:26 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63343/
> ---
> 
> (Updated Oct. 26, 2017, 9:26 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-17766
> https://issues.apache.org/jira/browse/HIVE-17766
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17766
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
> 3573d07e9a3bab4dabdc0b73292bd02ad05cbc89 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 46493ac006d00f2e6c0054da2438c0b01c1e3fd3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5ccb69ac4824a2002b2bddb1ad206cb7acd3ace3 
>   ql/src/test/queries/clientpositive/semijoin6.q PRE-CREATION 
>   ql/src/test/results/clientpositive/semijoin2.q.out 
> 2c3c5d8db3d5422f21fde020199e878b87b0507f 
>   ql/src/test/results/clientpositive/semijoin6.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63343/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 63343: HIVE-17766

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63343/
---

(Updated Oct. 26, 2017, 9:26 p.m.)


Review request for hive, Ashutosh Chauhan and Vineet Garg.


Bugs: HIVE-17766
https://issues.apache.org/jira/browse/HIVE-17766


Repository: hive-git


Description
---

HIVE-17766


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
3573d07e9a3bab4dabdc0b73292bd02ad05cbc89 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
46493ac006d00f2e6c0054da2438c0b01c1e3fd3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
5ccb69ac4824a2002b2bddb1ad206cb7acd3ace3 
  ql/src/test/queries/clientpositive/semijoin6.q PRE-CREATION 
  ql/src/test/results/clientpositive/semijoin2.q.out 
2c3c5d8db3d5422f21fde020199e878b87b0507f 
  ql/src/test/results/clientpositive/semijoin6.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/63343/diff/2/

Changes: https://reviews.apache.org/r/63343/diff/1-2/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Review Request 63343: HIVE-17766

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63343/
---

Review request for hive, Ashutosh Chauhan and Vineet Garg.


Bugs: HIVE-17766
https://issues.apache.org/jira/browse/HIVE-17766


Repository: hive-git


Description
---

HIVE-17766


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
3573d07e9a3bab4dabdc0b73292bd02ad05cbc89 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
46493ac006d00f2e6c0054da2438c0b01c1e3fd3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
5ccb69ac4824a2002b2bddb1ad206cb7acd3ace3 
  ql/src/test/queries/clientpositive/semijoin6.q PRE-CREATION 
  ql/src/test/results/clientpositive/semijoin2.q.out 
2c3c5d8db3d5422f21fde020199e878b87b0507f 
  ql/src/test/results/clientpositive/semijoin6.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/63343/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-17914) Provide support for complex expressions in ON clauses for SEMIjoins

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-17914:
--

 Summary: Provide support for complex expressions in ON clauses for 
SEMIjoins
 Key: HIVE-17914
 URL: https://issues.apache.org/jira/browse/HIVE-17914
 Project: Hive
  Issue Type: Bug
  Components: Operators, Parser
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Similar to HIVE-15251, HIVE-15211.

Currently, we have some restrictions on the predicates that we can use in ON 
clauses for semi joins. It is important for subquery decorrelation to support 
complex conditions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63144: HIVE-16748: Integreate YETUS to Pre-Commit

[Adam Szita via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63144/
---

(Updated Oct. 26, 2017, 7:04 p.m.)


Review request for hive, Peter Vary and Barna Zsombor Klara.


Changes
---

Addressed comments


Bugs: HIVE-16748
https://issues.apache.org/jira/browse/HIVE-16748


Repository: hive-git


Description
---

We already have Yetus check script ready for run in dev-support. We should 
integrate this with the automated ptest infrastructure so that during Precommit 
test we get a Yetus result back as Jira comment


Diffs (updated)
-

  dev-support/hive-personality.sh f3247aac6284b8dd863691b4819a10c3a896d50c 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
65a8216f6a076b0ee7baee11ca557f5e9f746316 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/YetusPhase.java 
PRE-CREATION 
  testutils/ptest2/src/main/resources/source-prep.vm 
7ad50248af02dfaeb6524a61d4895f1a8efba211 
  testutils/ptest2/src/main/resources/yetus-exec.vm PRE-CREATION 


Diff: https://reviews.apache.org/r/63144/diff/2/

Changes: https://reviews.apache.org/r/63144/diff/1-2/


Testing
---

Tested on Cloudera sponsored sandbox ptest server.


Thanks,

Adam Szita

[jira] [Created] (HIVE-17913) Cleanup unused methods in Driver

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-17913:
---

 Summary: Cleanup unused methods in Driver
 Key: HIVE-17913
 URL: https://issues.apache.org/jira/browse/HIVE-17913
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17912) org.apache.hadoop.hive.metastore.security.DBTokenStore - Parameterize Logging

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-17912:
--

 Summary: org.apache.hadoop.hive.metastore.security.DBTokenStore - 
Parameterize Logging
 Key: HIVE-17912
 URL: https://issues.apache.org/jira/browse/HIVE-17912
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17911) org.apache.hadoop.hive.metastore.ObjectStore - Tune Up

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-17911:
--

 Summary: org.apache.hadoop.hive.metastore.ObjectStore - Tune Up
 Key: HIVE-17911
 URL: https://issues.apache.org/jira/browse/HIVE-17911
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
Priority: Minor


# Remove unused variables
# Add logging parameterization
# Use CollectionUtils.isEmpty/isNotEmpty to simplify and unify collection empty 
check (and always use null check)
# Minor tweaks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17910) Hive does not build, missing files in classification sub-module.

[Harish Jaiprakash (JIRA)]

Harish Jaiprakash created HIVE-17910:


 Summary: Hive does not build, missing files in classification 
sub-module.
 Key: HIVE-17910
 URL: https://issues.apache.org/jira/browse/HIVE-17910
 Project: Hive
  Issue Type: Bug
Reporter: Harish Jaiprakash


hive does not build anymore for me, InterfaceAudience.java and 
InterfaceStability.java have been removed in HIVE-17743's commit. The patch 
looks fine, most likely a missed git add of classification. Please fix.

cc [~stakiar]. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17909) JDK9: TezAM fails to start

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-17909:
---

 Summary: JDK9: TezAM fails to start 
 Key: HIVE-17909
 URL: https://issues.apache.org/jira/browse/HIVE-17909
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)