Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

(Updated Jan. 5, 2019, 2:54 a.m.)


Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Changes
---

Added createTime in catalog as suggested.


Summary (updated)
-

HIVE-21077 : Database and Catalogs should have creation time


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description (updated)
---

HIVE-21077 : Database and Catalogs should have creation time


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
 3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 2953a8f327eabdee42dbc66e0c65f89d17add59a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
 e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestCatalogs.java
 28eb1fadca80dfd3c962e4163120b83f00410c4a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 


Diff: https://reviews.apache.org/r/69664/diff/2/

Changes: https://reviews.apache.org/r/69664/diff/1-2/


Testing (updated)
---

Ran ITests to check the db install and upgrade scripts are working for mysql, 
postgres, oracle and derby databases. The ITests for mssql is timing out for 
some reason due to container provisioning issues.


Thanks,

Vihang Karajgaonkar

[jira] [Created] (HIVE-21089) Automate dbinstall tests

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-21089:
--

 Summary: Automate dbinstall tests
 Key: HIVE-21089
 URL: https://issues.apache.org/jira/browse/HIVE-21089
 Project: Hive
  Issue Type: Sub-task
Reporter: Vihang Karajgaonkar


When a patch makes a schema change, precommit should run the dbinstall tests to 
make sure that the db scripts are working on all the supported databases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21088) Improve usability of dbinstall tests

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-21088:
--

 Summary: Improve usability of dbinstall tests
 Key: HIVE-21088
 URL: https://issues.apache.org/jira/browse/HIVE-21088
 Project: Hive
  Issue Type: Sub-task
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


There are nice integration tests which can be run manually for testing database 
schema changes. These tests spin up docker containers and install and upgrade 
the schema. Currently,  these tests expect that the host provides native 
support for docker daemon which is true in most cases. However, if you are 
using a lower version of macos (I tried it using 10.11), docker application 
cannot be installed and we need to install docker-toolbox instead. The issue 
with using docker-toolbox is that the docker daemon runs in a VM on the host 
which has a different IP address and hence the hardcoded {{localhost}} in the 
jdbc urls don't work. We can add a simple flag to provide the docker-machine ip 
as a  commandline arguemnt to override using localhost in the url.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69672: HIVE-21045: Add total API timing stats and connection pool stats to metrics

[Karthik Manamcheri via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69672/#review211698
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
Line 228 (original), 227 (patched)


I cleaned up this code a little bit.

The code in master had this logic where it checked if there are custom 
properties (through the function hasProviderSpecificConfigurations) which 
started with "hikaricp", "bonecp" etc. These custom properties should not 
matter becuase based on "datanuclues.connectionPoolType" property, we just 
create the appropriate pooling data source and pass in all the properties. 

New logic is that it looks at the connectionPoolingType property, checks if 
there is a DataSourceProvider avaialble for the set property and creates one. 
If not, we fallback to the default JDOHelper.getPersistenceManagerFactory logic.

Please correct me if I am wrong in my understanding here.


- Karthik Manamcheri


On Jan. 5, 2019, 12:35 a.m., Karthik Manamcheri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69672/
> ---
> 
> (Updated Jan. 5, 2019, 12:35 a.m.)
> 
> 
> Review request for hive, Adam Holley, Morio Ramdenbourg, Naveen Gangam, and 
> Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21045: Add total API timing stats and connection pool stats to metrics
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
>  dfd7abff85 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
>  7e33c519a8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
>  6dc63fb3bc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
>  5a92e104be 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
>  7fe487b184 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
>  8f6ae57e36 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/MetricsConstants.java
>  3b188f83af 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/PerfLogger.java
>  a2def26fc5 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  2a6290315a 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
>  6ae7f50471 
> 
> 
> Diff: https://reviews.apache.org/r/69672/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to verify that the new metrics show up for hikaricp, bonecp, 
> and also the total stats. Here are samples of
> 1. [HikariCP json metrics 
> sample](https://gist.github.com/kmanamcheri/48ff2a680e85c7e925a6f95a9384dcef)
> 2. [BoneCP json metrics 
> sample](https://gist.github.com/kmanamcheri/b005f68263a1a1be06b25156a159d975)
> 
> In both the reports note that there are pool gauges (for tracking the 
> connection pool info) and also a timer for total api calls.
> 
> 
> Thanks,
> 
> Karthik Manamcheri
> 
>

Review Request 69672: HIVE-21045: Add total API timing stats and connection pool stats to metrics

[Karthik Manamcheri via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69672/
---

Review request for hive, Adam Holley, Morio Ramdenbourg, and Naveen Gangam.


Repository: hive-git


Description
---

HIVE-21045: Add total API timing stats and connection pool stats to metrics


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
 dfd7abff85 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 7e33c519a8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
 6dc63fb3bc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 5a92e104be 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
 7fe487b184 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
 8f6ae57e36 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/MetricsConstants.java
 3b188f83af 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/PerfLogger.java
 a2def26fc5 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 2a6290315a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 6ae7f50471 


Diff: https://reviews.apache.org/r/69672/diff/1/


Testing
---

Manual testing to verify that the new metrics show up for hikaricp, bonecp, and 
also the total stats. Here are samples of
1. [HikariCP json metrics 
sample](https://gist.github.com/kmanamcheri/48ff2a680e85c7e925a6f95a9384dcef)
2. [BoneCP json metrics 
sample](https://gist.github.com/kmanamcheri/b005f68263a1a1be06b25156a159d975)

In both the reports note that there are pool gauges (for tracking the 
connection pool info) and also a timer for total api calls.


Thanks,

Karthik Manamcheri

[jira] [Created] (HIVE-21087) Query with filter on Coalesce returns wrong data

[Ravi Shetye (JIRA)]

Ravi Shetye created HIVE-21087:
--

 Summary: Query with filter on Coalesce returns wrong data
 Key: HIVE-21087
 URL: https://issues.apache.org/jira/browse/HIVE-21087
 Project: Hive
  Issue Type: Bug
  Components: Query Planning, Query Processor
Affects Versions: 3.1.0
Reporter: Ravi Shetye


QUERY

{code:java}
SELECT
 t_12.city_c14 c_4,
 REGEXP_EXTRACT( factinternetsales_t5.product_info, '(^|,)color:([^,]+)', 2 ),  
 COALESCE( t_12.city_c14 = 'Sydney' AND REGEXP_EXTRACT( 
factinternetsales_t5.product_info, '(^|,)color:([^,]+)', 2 ) = 'Black', false)
FROM
 (select 1 customerkey, "color:Red" product_info union select 2 customerkey, 
"color:Black" product_info  union select 3 customerkey, "color:Red" 
product_info  union select 4 customerkey, "color:Black" product_info  union 
select 5 customerkey, "color:Red" product_info )factinternetsales_t5
 JOIN (
  SELECT 1 customerkey_c10, 1 geographykey_c9 UNION SELECT 2 customerkey_c10, 1 
geographykey_c9 UNION SELECT 3 customerkey_c10, 1 geographykey_c9 UNION SELECT 
4 customerkey_c10, 2 geographykey_c9 UNION SELECT 5 customerkey_c10, 2 
geographykey_c9 UNION SELECT 1 customerkey_c10, 2 geographykey_c9
 ) t_8 ON factinternetsales_t5.customerkey = t_8.customerkey_c10
 JOIN ( SELECT 'Sydney' city_c14, 1 geographykey_c13 UNION SELECT 'Paris' 
city_c14, 2 geographykey_c13 
 ) t_12 ON t_8.geographykey_c9 = t_12.geographykey_c13
WHERE
 COALESCE( t_12.city_c14 = 'Sydney' AND REGEXP_EXTRACT( 
factinternetsales_t5.product_info, '(^|,)color:([^,]+)', 2 ) = 'Black', false)
GROUP BY
 t_12.city_c14,
 REGEXP_EXTRACT( factinternetsales_t5.product_info, '(^|,)color:([^,]+)', 2 ), 
 COALESCE( t_12.city_c14 = 'Sydney' AND REGEXP_EXTRACT( 
factinternetsales_t5.product_info, '(^|,)color:([^,]+)', 2 ) = 'Black', false); 
 
{code}

RESULT

{noformat}
+-+++
|   c_4   |  _c1   |  _c2   |
+-+++
| Paris   | Black  | false  |
| Paris   | Red| false  |
| Sydney  | Black  | true   |
| Sydney  | Red| false  |
+-+++
{noformat}

_c2 is the exact expression in where, so the rows which have entry false should 
not be present in the output




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69664: HIVE-21077 : Database should have creation time

[Karthik Manamcheri via Review Board]

> On Jan. 4, 2019, 5:33 p.m., Karthik Manamcheri wrote:
> >
> 
> Karthik Manamcheri wrote:
> 1. While we are at it, can we make sure that the "create time" exists for 
> everything else (if required)? Such as catalog, partitions..
> 2. Out of curiosity.. are there any tests to ensure that the upgrade path 
> works? What if we miss adding this to some sql file? Would we catch it?
> 
> Vihang Karajgaonkar wrote:
> createTime exists for tables and partitions. It doesn't exist for 
> catalog. I will see if it doesn't add too much to this patch to add 
> createTime to catalogs. Otherwise, I will prefer to do it in a separate 
> patch. For upgrade testing the precommit job does not run such tests. It is 
> expected that schema patches are tested either manually or using the ITests 
> (steps are provided in DEV-README). I have tested this patch on mysql 
> manually and I am setting up my machine to give a try to the newly introduced 
> ITests currently. Will update the RB once I complete the testing.

Don't you think its better to make the schema changes all together instead of 
splitting it up. If we think we might need creation time for catalogs, we 
should add it to this changeset because we can knock off the schema change as 
well.


- Karthik


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/#review211677
---


On Jan. 3, 2019, 11:44 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69664/
> ---
> 
> (Updated Jan. 3, 2019, 11:44 p.m.)
> 
> 
> Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.
> 
> 
> Bugs: HIVE-21077
> https://issues.apache.org/jira/browse/HIVE-21077
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21077 : Database should have creation time
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
>  994797698a379e0b08604d73d2d6728a2fcee4df 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  2953a8f327eabdee42dbc66e0c65f89d17add59a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d43c0c1e70cffbebd39b05f89ec396227c58ac77 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
>  f3d2182a04ab81417a4ba58d9340721513e8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
>  815b39c483b2233660310983d58194fb1ab2d107 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> caaec457194332a99d5cd57bef746e969dd38161 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  a3c4196dbff7e53be5317631b314983d16a99020 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  bcaebd18accf86846ae44a6498046514575fc069 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  edde08db9ef7ee01800c7cc3a04c813014abdd18 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  701acb00984c61f7511dcc48053890b154575d1f 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  b1980c5b83f16614845063516495188ebdd8c2a3 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  b9f63313251ab1fa6278b862ed9e07e62b234c04 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  

Re: Review Request 69664: HIVE-21077 : Database should have creation time

[Vihang Karajgaonkar via Review Board]

> On Jan. 4, 2019, 5:33 p.m., Karthik Manamcheri wrote:
> >
> 
> Karthik Manamcheri wrote:
> 1. While we are at it, can we make sure that the "create time" exists for 
> everything else (if required)? Such as catalog, partitions..
> 2. Out of curiosity.. are there any tests to ensure that the upgrade path 
> works? What if we miss adding this to some sql file? Would we catch it?

createTime exists for tables and partitions. It doesn't exist for catalog. I 
will see if it doesn't add too much to this patch to add createTime to 
catalogs. Otherwise, I will prefer to do it in a separate patch. For upgrade 
testing the precommit job does not run such tests. It is expected that schema 
patches are tested either manually or using the ITests (steps are provided in 
DEV-README). I have tested this patch on mysql manually and I am setting up my 
machine to give a try to the newly introduced ITests currently. Will update the 
RB once I complete the testing.


> On Jan. 4, 2019, 5:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
> > Lines 785 (patched)
> > 
> >
> > Why not just store it in milliseconds?

This is to be consitent with the createTime stored in table and partitions


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/#review211677
---


On Jan. 3, 2019, 11:44 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69664/
> ---
> 
> (Updated Jan. 3, 2019, 11:44 p.m.)
> 
> 
> Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.
> 
> 
> Bugs: HIVE-21077
> https://issues.apache.org/jira/browse/HIVE-21077
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21077 : Database should have creation time
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
>  994797698a379e0b08604d73d2d6728a2fcee4df 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  2953a8f327eabdee42dbc66e0c65f89d17add59a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d43c0c1e70cffbebd39b05f89ec396227c58ac77 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
>  f3d2182a04ab81417a4ba58d9340721513e8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
>  815b39c483b2233660310983d58194fb1ab2d107 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> caaec457194332a99d5cd57bef746e969dd38161 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  a3c4196dbff7e53be5317631b314983d16a99020 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  bcaebd18accf86846ae44a6498046514575fc069 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  edde08db9ef7ee01800c7cc3a04c813014abdd18 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  701acb00984c61f7511dcc48053890b154575d1f 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  b1980c5b83f16614845063516495188ebdd8c2a3 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  b9f63313251ab1fa6278b862ed9e07e62b234c04 
>   
> 

Re: Review Request 69664: HIVE-21077 : Database should have creation time

[Karthik Manamcheri via Review Board]

> On Jan. 4, 2019, 5:33 p.m., Karthik Manamcheri wrote:
> >

1. While we are at it, can we make sure that the "create time" exists for 
everything else (if required)? Such as catalog, partitions..
2. Out of curiosity.. are there any tests to ensure that the upgrade path 
works? What if we miss adding this to some sql file? Would we catch it?


- Karthik


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/#review211677
---


On Jan. 3, 2019, 11:44 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69664/
> ---
> 
> (Updated Jan. 3, 2019, 11:44 p.m.)
> 
> 
> Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.
> 
> 
> Bugs: HIVE-21077
> https://issues.apache.org/jira/browse/HIVE-21077
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21077 : Database should have creation time
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
>  994797698a379e0b08604d73d2d6728a2fcee4df 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  2953a8f327eabdee42dbc66e0c65f89d17add59a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d43c0c1e70cffbebd39b05f89ec396227c58ac77 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
>  f3d2182a04ab81417a4ba58d9340721513e8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
>  815b39c483b2233660310983d58194fb1ab2d107 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> caaec457194332a99d5cd57bef746e969dd38161 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  a3c4196dbff7e53be5317631b314983d16a99020 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  bcaebd18accf86846ae44a6498046514575fc069 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  edde08db9ef7ee01800c7cc3a04c813014abdd18 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  701acb00984c61f7511dcc48053890b154575d1f 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  b1980c5b83f16614845063516495188ebdd8c2a3 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  b9f63313251ab1fa6278b862ed9e07e62b234c04 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  0c36069d071d4b60cc338ba729da5d22e08ca8ca 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
>  d323ac6c90ed20f092b4e179fdb1bed8602ecf63 
> 
> 
> Diff: https://reviews.apache.org/r/69664/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69664: HIVE-21077 : Database should have creation time

[Karthik Manamcheri via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/#review211677
---




standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 383 (patched)


nit: add comment saying that the unti of this is "seconds since epoch"



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 785 (patched)


Why not just store it in milliseconds?


- Karthik Manamcheri


On Jan. 3, 2019, 11:44 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69664/
> ---
> 
> (Updated Jan. 3, 2019, 11:44 p.m.)
> 
> 
> Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.
> 
> 
> Bugs: HIVE-21077
> https://issues.apache.org/jira/browse/HIVE-21077
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21077 : Database should have creation time
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
>  994797698a379e0b08604d73d2d6728a2fcee4df 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  2953a8f327eabdee42dbc66e0c65f89d17add59a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d43c0c1e70cffbebd39b05f89ec396227c58ac77 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
>  f3d2182a04ab81417a4ba58d9340721513e8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
>  815b39c483b2233660310983d58194fb1ab2d107 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> caaec457194332a99d5cd57bef746e969dd38161 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  a3c4196dbff7e53be5317631b314983d16a99020 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  bcaebd18accf86846ae44a6498046514575fc069 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  edde08db9ef7ee01800c7cc3a04c813014abdd18 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  701acb00984c61f7511dcc48053890b154575d1f 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  b1980c5b83f16614845063516495188ebdd8c2a3 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  b9f63313251ab1fa6278b862ed9e07e62b234c04 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  0c36069d071d4b60cc338ba729da5d22e08ca8ca 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
>  d323ac6c90ed20f092b4e179fdb1bed8602ecf63 
> 
> 
> Diff: https://reviews.apache.org/r/69664/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69585: HIVE-20776: Run HMS filterHooks on server-side in addition to client-side

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69585/
---

(Updated Jan. 4, 2019, 4:07 p.m.)


Review request for hive, Adam Holley, Morio Ramdenbourg, Peter Vary, Sergio 
Pena, and Vihang Karajgaonkar.


Bugs: HIVE-20776
https://issues.apache.org/jira/browse/HIVE-20776


Repository: hive-git


Description
---

add filtering to read result at HMS server, so user cannot see metadata he/she 
has no privileges. Filtering is enabled/disabled based on configuration.


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 748b56b0a268c1ec7dea022722478ec50889c016 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 be1f8c78497fe3d0816ad3935ba07cd5ad379b08 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FilterUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestFilterHooks.java
 7dc69bc4e92875c8962dcd313b16f0f90ea8b057 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetastoreFilterHook.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69585/diff/7/

Changes: https://reviews.apache.org/r/69585/diff/6-7/


Testing
---

Existing unit tests passed. 
add new unit tests for filtering at HMS server and HMS client
add code to enabled/disable filtering at HMS client based on configuration


Thanks,

Na Li

[jira] [Created] (HIVE-21086) Fetch task and MR job give back different results when multiple partitions point to the same location

[Laszlo Pinter (JIRA)]

Laszlo Pinter created HIVE-21086:


 Summary: Fetch task and MR job give back different results when 
multiple partitions point to the same location
 Key: HIVE-21086
 URL: https://issues.apache.org/jira/browse/HIVE-21086
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 4.0.0
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
 Fix For: 4.0.0


Create a table with 2 partitions where both partitions share the same location 
and inserting a single line to one of them.
{code:sql}
create table test (i int) partitioned by (j int);
alter table test add partition (j=1);
alter table test add partition (j=2) location '<>';
insert into table test partition (j=1) values (1);{code}
Make sure *hive.fetch.task.conversion=more*.
{code:sql}
set hive.fetch.task.conversion=more;{code}
The result of a simple select * test:

+---+---+
|test.i |test.j |

+---+---+
|1      |1      |
|1      |2      |

+---+---+

Change the *hive.fetch.task.conversion=none;*
{code:sql}
set hive.fetch.task.conversion=none;
{code}
The result of a simple select * test:
 +---+---+
|test.i |test.j |

+---+---+
|1      |2      |

+---+---+



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69642: HIVE-20977: Lazy evaluate the table object in PreReadTableEvent to improve get_partition performance

[Peter Vary via Review Board]

> On jan. 3, 2019, 1:42 de, Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetPartitions.java
> > Lines 374 (patched)
> > 
> >
> > The behavior of the getPartitions\* changed to be more inline with how 
> > the other getTable/getDatabase calls work.
> > 
> > Before this change, if you issue a getPartitionsByNames with an empty 
> > database, we threw an exception. After this change, we will return an empty 
> > list of partitions instead. This behavior is similar to what happens if you 
> > issue a getTablesByNames call (an empty list of tables are returned)
> 
> Peter Vary wrote:
> This change is also worrying. This is also an API change which might 
> cause backward incompatibility problems with customers expecting empty list.
> We proposed this kind of changes a way back, and the community consensus 
> was that we should not change even the Exception types that have been thrown.
> 
> Karthik Manamcheri wrote:
> Actually I am fixing a regression bug. In the master branch (without my 
> change), if you remove the TransactionalValidationListener from the list of 
> pre-listeners, the GetPartitions API tests fail! The getPartitions API test 
> depends on the fact there is a a pre-event listener. Customers already expect 
> the API to throw an exception (or return an empty list) depending on if there 
> is a listener attached or not. So the API contract itself is that it can 
> throw an exception (OR return an empty list). This is how earlier versions of 
> Hive are. HIVE-12064 introduced a bug which changed the behavior of clients. 
> My change makes it so that it behaves similar to how earlier versions of Hive 
> behaved (before HIVE-12064).
> 
> We should also fix the tests so that they don't depend of the existence 
> (or non-existence) of listeners or plugins.
> 
> Basically what I am saying is that this API change is not new and is how 
> Hive 1.x/2.x behaved (before HIVE-12064). We wrote the unit tests around the 
> bug!

Bugs which are exist for too long prone to become features :D
I personally agree with you. but I would be more confortible if we could find 1 
more committer to validate our view - maybe Alan Gates, Thejas Nair who were 
the ones who actively participated in our "IMetaStoreClient and HMS Thrift API 
exception handling" discussion. You might want to ping them.

Thanks,
Peter


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69642/#review211625
---


On jan. 3, 2019, 1:40 de, Karthik Manamcheri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69642/
> ---
> 
> (Updated jan. 3, 2019, 1:40 de)
> 
> 
> Review request for hive, Adam Holley, Na Li, Morio Ramdenbourg, Naveen 
> Gangam, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20977: Lazy evaluate the table object in PreReadTableEvent to improve 
> get_partition performance
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/PreReadTableEvent.java
>  beec72bc12 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/ThrowingSupplier.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
>  7429d18226 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
>  fe64a91b56 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetPartitions.java
>  4d7f7c1220 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
>  a338bd4032 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/events/TestPreReadTableEvent.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69642/diff/4/
> 
> 
> Testing
> ---
> 
> Unit tests.
> Manual performance test with Cloudera BDR to notice improved backup 
> performance.
> 
> 
> Thanks,
> 
> Karthik Manamcheri
> 
>

[jira] [Created] (HIVE-21085) Materialized views registry starts non-external tez session

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-21085:
--

 Summary: Materialized views registry starts non-external tez 
session
 Key: HIVE-21085
 URL: https://issues.apache.org/jira/browse/HIVE-21085
 Project: Hive
  Issue Type: Bug
  Components: Materialized views
Reporter: Prasanth Jayachandran
Assignee: Jesus Camacho Rodriguez


Materialized views registry is doing SessionState.start() which will start 
regular tez session. In the presence of external tez sessions, it should get 
session from external sessions pool. There are also other places where 
SessionState.start() might be invoked.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)