Re: Review Request 68630: HIVE-20420: Provide a fallback authorizer when no other authorizer is in use

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68630/
---

(Updated Sept. 11, 2018, 11:19 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-20420


Diffs (updated)
-

  ql/pom.xml a55cbe3 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/SettableConfigUpdater.java
 12be41c 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/fallback/FallbackHiveAuthorizerFactory.java
 PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_addjar.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_compile.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_create_func2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_dfs.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_disallow_transform.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_load.q PRE-CREATION 
  ql/src/test/queries/clientnegative/fallbackauth_set_invalidconf.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_addjar.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_compile.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func1.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_create_func2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_dfs.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_disallow_transform.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_load.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/fallbackauth_set_invalidconf.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68630/diff/2/

Changes: https://reviews.apache.org/r/68630/diff/1-2/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-20540) Vectorization : Support loading bucketed tables using sorted dynamic partition optimizer - II

[Deepak Jaiswal (JIRA)]

Deepak Jaiswal created HIVE-20540:
-

 Summary: Vectorization : Support loading bucketed tables using 
sorted dynamic partition optimizer - II
 Key: HIVE-20540
 URL: https://issues.apache.org/jira/browse/HIVE-20540
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


Followup to HIVE-20510 with remaining issues,

 

1. Avoid using Reflection.
2. In VectorizationContext, use correct place to setup the VectorExpression. It 
may be missed in certain cases.
3. In BucketNumExpression, make sure that a value is not overwritten before it 
is processed. Use a flag to achieve this.

cc [~gopalv]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68664: HIVE-20306: Implement projection spec for fetching only requested fields from partitions

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68664/
---

(Updated Sept. 11, 2018, 9:39 p.m.)


Review request for hive, Aihua Xu, Peter Vary, Todd Lipcon, and Vihang 
Karajgaonkar.


Changes
---

Incorporated code review feedback from akolb


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306: Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 ae0956870a7d01c24f5fdaa07094c3dc6604ab9a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 4574c6a4925ae3df9dd1ee7b8786976ae6fc8397 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 22deffe1d31a64f95c49d7f017dfeb2994233e71 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 38074ce79b8a06b3795d00431025240778abb569 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 38fac465d73c264f85fc512548ebe1919ee35c17 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 0192c6da314694c1253b49949bbe749902f49b4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 e6a72762bb7b0d36fdf6d20d02cb1da3337a98a0 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
85a5c601e03ecd2fb6ac5d30d789193e10bf38c2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 ba82a9327cf18e8d55ebddcd774786d3d72f753a 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 df6d56b679a03f88fe0de048d7b2d7e47709996c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 54e7eda0da796877f1331de137d534126375c6ba 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java
 4c6ce008f89469353bfee3175168a518534a42b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 10ff9dfbb6d8f61fa75f731f4cd0f006c98e0067 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 c681a87a1c6b10a4f9494e49a42282cf90027ad7 
  

[jira] [Created] (HIVE-20539) Remove dependency on com.metamx.java-util

[Nishant Bangarwa (JIRA)]

Nishant Bangarwa created HIVE-20539:
---

 Summary: Remove dependency on com.metamx.java-util
 Key: HIVE-20539
 URL: https://issues.apache.org/jira/browse/HIVE-20539
 Project: Hive
  Issue Type: Task
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


java-util was moved from com.metamx to druid code repository. 
Currently we are packing both com.metamx.java-jtil and io.druid.java-util, 
This task is to remove the dependency on com.metamx.java-util



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20538) Allow to store a key value together with a transaction.

[Jaume M (JIRA)]

Jaume M created HIVE-20538:
--

 Summary: Allow to store a key value together with a transaction.
 Key: HIVE-20538
 URL: https://issues.apache.org/jira/browse/HIVE-20538
 Project: Hive
  Issue Type: New Feature
  Components: Standalone Metastore, Transactions
Reporter: Jaume M


This can be useful for example to know if a transaction has already happened.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20537) Multi-column joins estimates with uncorrelated columns different in CBO and Hive

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-20537:
--

 Summary: Multi-column joins estimates with uncorrelated columns 
different in CBO and Hive
 Key: HIVE-20537
 URL: https://issues.apache.org/jira/browse/HIVE-20537
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20536) Add Surrogate Keys function to Hive

[Miklos Gergely (JIRA)]

Miklos Gergely created HIVE-20536:
-

 Summary: Add Surrogate Keys function to Hive
 Key: HIVE-20536
 URL: https://issues.apache.org/jira/browse/HIVE-20536
 Project: Hive
  Issue Type: Task
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely


Surrogate keys is an ability to generate and use unique integers for each row 
in a table. If we have that ability then in conjunction with default clause we 
can get surrogate keys functionality. Consider following ddl:

create table t1 (a string, b bigint default unique_long());

We already have default clause wherein you can specify a function to provide 
values. So, what we need is udf which can generate unique longs for each row 
across queries for a table. 
Idea is to use write_id . This is a column in metastore table TXN_COMPONENTS 
whose value is determined at compile time to be used during query execution. 
Each query execution generates a new write_id. So, we can seed udf with this 
value during compilation.
Then we statically allocate ranges for each task from which it can draw next 
long. So, lets say 64-bit write_id we divy up such that last 24 bits belong to 
original usage of it that is txns. Next 16 bits are used for task_attempts and 
last 24 bits to generate new long for each row. This implies we can allow 17M 
txns, 65K tasks and 17M rows in a task. If you hit any of those limits we can 
fail the query.

Implementation wise: serialize write_id in initialize() of udf. Then during 
execute() we find out what task_attempt current task is and use it along with 
write_id() to get starting long and give a new value on each invocation of 
execute().

Here we are assuming write_id can be determined at compile time, which should 
be the case but we need to figure out how to get handle to it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #431: HIVE-20531 : One of the task , either move or add pa...

[maheshk114]

GitHub user maheshk114 opened a pull request:

https://github.com/apache/hive/pull/431

HIVE-20531 : One of the task , either move or add partition can be avoided 
in repl load flow

…

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maheshk114/hive HIVE-20531

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/431.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #431


commit a7c693696bd961937db3a3594c4717b4f180e48e
Author: Mahesh Kumar Behera 
Date:   2018-09-11T11:25:26Z

HIVE-20531 : One of the task , either move or add partition can be avoided 
in repl load flow




---

Speakers needed for Apache DC Roadshow

[Rich Bowen]

We need your help to make the Apache Washington DC Roadshow on Dec 4th a 
success.


What do we need most? Speakers!

We're bringing a unique DC flavor to this event by mixing Open Source 
Software with talks about Apache projects as well as OSS CyberSecurity, 
OSS in Government and and OSS Career advice.


Please take a look at: http://www.apachecon.com/usroadshow18/

(Note: You are receiving this message because you are subscribed to one 
or more mailing lists at The Apache Software Foundation.)


Rich, for the ApacheCon Planners

--
rbo...@apache.org
http://apachecon.com
@ApacheCon

Review Request 68683: Add new configuration to set the size of the global compile lock

[denys kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68683/
---

Review request for hive, Zoltan Haindrich, Zoltan Haindrich, and Peter Vary.


Bugs: HIVE-20535
https://issues.apache.org/jira/browse/HIVE-20535


Repository: hive-git


Description
---

When removing the compile lock, it is quite risky to remove it entirely.

It would be good to provide a pool size for the concurrent compilation, so the 
administrator can limit the load


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
 af0f87bac3 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8c39de3e77 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 737debd2ad 
  ql/src/test/org/apache/hadoop/hive/ql/CompileLockTest.java PRE-CREATION 


Diff: https://reviews.apache.org/r/68683/diff/1/


Testing
---

Added CompileLockTest


Thanks,

denys kuzmenko

[jira] [Created] (HIVE-20535) Add new configuration to set the size of the global compile lock

[denys kuzmenko (JIRA)]

denys kuzmenko created HIVE-20535:
-

 Summary: Add new configuration to set the size of the global 
compile lock
 Key: HIVE-20535
 URL: https://issues.apache.org/jira/browse/HIVE-20535
 Project: Hive
  Issue Type: Task
  Components: HiveServer2
Reporter: denys kuzmenko


When removing the compile lock, it is quite risky to remove it entirely.

It would be good to provide a pool size for the concurrent compilation, so the 
administrator can limit the load



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68525: HIVE-20296 Improve HivePointLookupOptimizerRule to be able to extract from more sophisticated contexts

[Zoltan Haindrich]

> On Sept. 6, 2018, 1:39 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
> > Lines 278 (patched)
> > 
> >
> > Does this class also exist in Calcite? If so, can you please leave a 
> > note here to remove from Hive once these are made public in Calcite?

no; a but there is something similar inside RexSimplify - but those are private 
classes

I think it would be good to move this to Calcite.


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68525/#review208392
---


On Aug. 27, 2018, 4:01 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68525/
> ---
> 
> (Updated Aug. 27, 2018, 4:01 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-20296
> https://issues.apache.org/jira/browse/HIVE-20296
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> generalized rule to extract INs from more complex filter conditions as well.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
>  eff9a312aaa09a889f0c6a045bd813d1fb633956 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/TestHivePointLookupOptimizerRule.java
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/druid/druidmini_test_ts.q.out 
> a8e6894a9786318ed4362ee37d918a3699f074a0 
>   ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 
> 260ba1cbddee7f0946f0cdec1070359ab2a1d2aa 
>   ql/src/test/results/clientpositive/perf/spark/query15.q.out 
> 67684f6b0bc44c0cae6107be94d131a083eca0e1 
>   ql/src/test/results/clientpositive/perf/spark/query47.q.out 
> 690b1054c12f7d588015afc301802d9d2da2d0b9 
>   ql/src/test/results/clientpositive/perf/spark/query57.q.out 
> 51e644a87bf4befbe4368cdf03b5eaab6d4f2049 
>   ql/src/test/results/clientpositive/perf/tez/query15.q.out 
> e1eca99d95e13070f901b75a406391435b2b4f1d 
>   ql/src/test/results/clientpositive/perf/tez/query47.q.out 
> d034ea9433a3b1c54c545edf526c215bf79388e1 
>   ql/src/test/results/clientpositive/perf/tez/query57.q.out 
> 42cbbdc2a4d8bc469c9d91867faf21fc94a057ea 
> 
> 
> Diff: https://reviews.apache.org/r/68525/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>