Re: Review Request 61009: Extend object store to store bit vectors

2017-07-24 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/
---

(Updated July 25, 2017, 1:03 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16997


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
  
common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
 e810ac5487 
  common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
d1955468a6 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ebe503083 
  
common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
 PRE-CREATION 
  data/conf/hive-site.xml 62364fe4ea 
  data/conf/llap/hive-site.xml cac5a3b775 
  data/conf/perf-reg/hive-site.xml 012369fdd1 
  data/conf/tez/hive-site.xml 28abc2dd92 
  metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
  metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
  metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
  metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
  metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
  metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
  metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
  metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 0a70d47cca 
  metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 1cdeb6b45a 
  metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
c44dd067fc 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
a960b2d26b 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
b52c94c9fb 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
db4ec91cdb 
  metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
2dc2804343 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
cea94a0d5a 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
0e119896a5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
 d81d612e92 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
 e796df2422 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
 29a05390bf 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
 568bf0609b 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
 8eb64e0143 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
 b6b86123b2 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java
 af75bced72 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
 2da6f60167 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
 83c6c54fd2 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BinaryColumnStatsMerger.java
 af0669eb65 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BooleanColumnStatsMerger.java
 33ff6a19f5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
 d3051a2b00 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 c013ba5c5d 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 e899bfe85f 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DecimalColumnStatsMerger.java
 4099ffcace 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DoubleColumnStatsMerger.java
 1691fc97df 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/LongColumnStatsMerger.java
 361af350fe

Re: [VOTE] Shall we release Hive 2.2.0 rc1?

2017-07-24 Thread Ashutosh Chauhan 
+1 built from sources. Ran few unit tests and queries. Looks good.

Thanks,
Ashutosh

On Mon, Jul 24, 2017 at 3:02 PM, Alan Gates  wrote:

> +1, checked the signatures, did a build with a clean repo, ran RAT.
>
> Alan.
>
> On Fri, Jul 21, 2017 at 5:20 PM, Owen O'Malley 
> wrote:
>
> > Ok, I rolled a new RC with the Apache header problem fixed.
> >
> > Artifacts: https://home.apache.org/~omalley/hive-2.2.0/
> > Tag: https://github.com/apache/hive/releases/tag/release-2.2.0rc1
> da840b0
> >
> > .. Owen
> >
>

Re: Review Request 61009: Extend object store to store bit vectors

2017-07-24 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/#review181299
---




metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql
Line 1 (original), 1 (patched)


change it to varbinary(max) since otherwise its limited to 8K bytes.



metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql
Line 1 (original), 1 (patched)


change it to varbinary(max) since otherwise its limited to 8K bytes.



metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql
Line 90 (original), 90 (patched)


change it to varbinary(max) since otherwise its limited to 8K bytes.



metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql
Line 1 (original), 1 (patched)


pg doesnt have blob type. Use bytea.



metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql
Line 537 (original), 537 (patched)


pg doesnt have blob type. Use bytea.



data/conf/hive-site.xml
Lines 317-320 (patched)


Can you also add this for:
data/conf/llap/hive-site.xml
data/conf/tez/hive-site.xml
data/conf/perf-reg/hive-site.xml ?


- Ashutosh Chauhan


On July 24, 2017, 11:32 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61009/
> ---
> 
> (Updated July 24, 2017, 11:32 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16997
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  e810ac5487 
>   common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> d1955468a6 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ebe503083 
>   
> common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
>  PRE-CREATION 
>   data/conf/hive-site.xml 62364fe4ea 
>   metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
>   metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
>   metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
>   metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
>   metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
>   metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
>   metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
>   metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
> 0a70d47cca 
>   metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 1cdeb6b45a 
>   metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
> c44dd067fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> a960b2d26b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> b52c94c9fb 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> db4ec91cdb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> 2dc2804343 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> cea94a0d5a 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 0e119896a5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
>  d81d612e92 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
>  e796df2422 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  29a05390bf 
>   
>

[jira] [Created] (HIVE-17161) BeeLine up arrow key should show the last sql instead of last line

2017-07-24 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-17161:
--

 Summary: BeeLine up arrow key should show the last sql instead of 
last line
 Key: HIVE-17161
 URL: https://issues.apache.org/jira/browse/HIVE-17161
 Project: Hive
  Issue Type: Improvement
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Currently, when you press Up arrow on beeline prompt, it shows the last line of 
the previous sql. It is hard to execute the previous sql if it was executed 
using multiple lines. It would be good to improve this experience by fetching 
the last command executed instead of just the last line of the previous command.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Apache ptest broken?

2017-07-24 Thread Deepak Jaiswal 
Looks like ptest for hive is broken.

https://builds.apache.org/view/H-L/view/Hive/job/PreCommit-HIVE-Build/#

Regards,
Deepak

Re: Review Request 61009: Extend object store to store bit vectors

2017-07-24 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/
---

(Updated July 24, 2017, 11:32 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16997


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
  
common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
 e810ac5487 
  common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
d1955468a6 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ebe503083 
  
common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
 PRE-CREATION 
  data/conf/hive-site.xml 62364fe4ea 
  metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
  metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
  metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
  metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
  metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
  metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
  metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
  metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 0a70d47cca 
  metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 1cdeb6b45a 
  metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
c44dd067fc 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
a960b2d26b 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
b52c94c9fb 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
db4ec91cdb 
  metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
2dc2804343 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
cea94a0d5a 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
0e119896a5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
 d81d612e92 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
 e796df2422 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
 29a05390bf 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
 568bf0609b 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
 8eb64e0143 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
 b6b86123b2 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java
 af75bced72 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
 2da6f60167 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
 83c6c54fd2 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BinaryColumnStatsMerger.java
 af0669eb65 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BooleanColumnStatsMerger.java
 33ff6a19f5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
 d3051a2b00 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 c013ba5c5d 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 e899bfe85f 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DecimalColumnStatsMerger.java
 4099ffcace 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DoubleColumnStatsMerger.java
 1691fc97df 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/LongColumnStatsMerger.java
 361af350fe 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/StringColumnStatsMerger.java
 8e28f907ee

Re: Review Request 61009: Extend object store to store bit vectors

2017-07-24 Thread pengcheng xiong 


> On July 21, 2017, 10:45 p.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 1507-1508 (original)
> > 
> >
> > Seems like we are going to loose this extrapolation logic.

it is back in the new version.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/#review181146
---


On July 24, 2017, 6:08 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61009/
> ---
> 
> (Updated July 24, 2017, 6:08 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16997
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  e810ac5487 
>   common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> d1955468a6 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java df45f2cc32 
>   
> common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
>   metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
>   metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
>   metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
>   metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
>   metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
>   metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
>   metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
> 0a70d47cca 
>   metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 1cdeb6b45a 
>   metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
> c44dd067fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> a960b2d26b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> b52c94c9fb 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> db4ec91cdb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> 2dc2804343 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 3ac4fe1604 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 0e119896a5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
>  d81d612e92 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
>  e796df2422 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  29a05390bf 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
>  568bf0609b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  8eb64e0143 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  b6b86123b2 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java
>  af75bced72 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  2da6f60167 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
>  83c6c54fd2 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BinaryColumnStatsMerger.java
>  af0669eb65 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BooleanColumnStatsMerger.java
>  33ff6a19f5 
>   
>

Re: [DISCUSS] Separating out the metastore as its own TLP

2017-07-24 Thread Gopal Vijayaraghavan 
Hi,


Changing the name isn't really optional or "being google-able" [2].

The naming is a crucial part of trademark protection [1], which is the only 
protection ASF has against hostile Embrace & Extends.

Fragmented forks with the same name is particularly bad, especially if the 
feature in question can be only used by a proprietary tool (like Dain's 
suggestion about Presto view metadata, except it only works with a per-cpu 
license).

The safe path isn't pretty, it still ends up with IcedTea and IceWeasel … but 
at least, those are clearly weird.

Cheers,
Gopal

[1] - https://en.wikipedia.org/wiki/A_moron_in_a_hurry#United_States
[2] - https://packages.debian.org/jessie/misc/metastore

On 7/24/17, 3:04 PM, "Carl Steinbach"  wrote:

+1 to Vihang's suggestion. Changing the name will only cause confusion.

On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera 
wrote:

> +1 Vihang, I do not really like Catalog as it could create confusion with
> the Catalog daemon from impala.
>
> On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar 
> wrote:
>
> > Before we see a flood of name suggestions :) Why not just keep it
> > Metastore? Its already well-known in the community and easy to relate 
to.
> >
> > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates 
> wrote:
> >
> > > In the same vein Carter and Gunther suggested Omegastore.  Pick your
> > > alphabet and whether it’s a catalog or a store I guess.
> > >
> > > Alan.
> > >
> > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > ser...@hortonworks.com>
> > > wrote:
> > >
> > > > I’d like to suggest ZCatalog.
> > > >
> > > > On 17/7/11, 15:41, "Lefty Leverenz"  wrote:
> > > >
> > > > >>> I'd like to suggest Riven.  (Owen O'Malley)
> > > > >
> > > > >> How about "Flora"?  (Andrew Sherman)
> > > > >
> > > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > > >
> > > > >Along the same lines, how about "Honeycomb"?
> > > > >
> > > > >But since the idea is to make the metastore useful for many
> projects,
> > a
> > > > >generic name that starts with "Meta" would be less confusing ...
> even
> > > > >though it breaks the tradition of Apache projects having quirky
> names.
> > > > >Unfortunately "Metalog" is already in use.  "Metamorph" has other
> > > > >connotations, but it's cool.
> > > > >
> > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of 
splitting
> > off
> > > > >the metastore into a new project:  -0.5 for the sake of Hive and
> +0.5
> > > for
> > > > >the greater good.  Wishy-washy, that's me.
> > > > >
> > > > >-- Lefty
> > > > >
> > > > >
> > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > asher...@cloudera.com>
> > > > >wrote:
> > > > >
> > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > owen.omal...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun 
> > > wrote:
> > > > >> >
> > > > >> > > and maybe a different project name?
> > > > >> > >
> > > > >> >
> > > > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > > > >> >
> > > > >> > .. Owen
> > > > >> >
> > > > >>
> > > > >> How about "Flora"?
> > > > >>
> > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > >>
> > > > >> -Andrew
> > > > >>
> > > >
> > > >
> > >
> >
>
>
>
> --
> - JRB
>

Re: [DISCUSS] Separating out the metastore as its own TLP

2017-07-24 Thread Carl Steinbach 
+1 to Vihang's suggestion. Changing the name will only cause confusion.

On Mon, Jul 24, 2017 at 2:28 PM, Johndee Cloudera 
wrote:

> +1 Vihang, I do not really like Catalog as it could create confusion with
> the Catalog daemon from impala.
>
> On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar 
> wrote:
>
> > Before we see a flood of name suggestions :) Why not just keep it
> > Metastore? Its already well-known in the community and easy to relate to.
> >
> > On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates 
> wrote:
> >
> > > In the same vein Carter and Gunther suggested Omegastore.  Pick your
> > > alphabet and whether it’s a catalog or a store I guess.
> > >
> > > Alan.
> > >
> > > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> > ser...@hortonworks.com>
> > > wrote:
> > >
> > > > I’d like to suggest ZCatalog.
> > > >
> > > > On 17/7/11, 15:41, "Lefty Leverenz"  wrote:
> > > >
> > > > >>> I'd like to suggest Riven.  (Owen O'Malley)
> > > > >
> > > > >> How about "Flora"?  (Andrew Sherman)
> > > > >
> > > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > > >
> > > > >Along the same lines, how about "Honeycomb"?
> > > > >
> > > > >But since the idea is to make the metastore useful for many
> projects,
> > a
> > > > >generic name that starts with "Meta" would be less confusing ...
> even
> > > > >though it breaks the tradition of Apache projects having quirky
> names.
> > > > >Unfortunately "Metalog" is already in use.  "Metamorph" has other
> > > > >connotations, but it's cool.
> > > > >
> > > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting
> > off
> > > > >the metastore into a new project:  -0.5 for the sake of Hive and
> +0.5
> > > for
> > > > >the greater good.  Wishy-washy, that's me.
> > > > >
> > > > >-- Lefty
> > > > >
> > > > >
> > > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> > asher...@cloudera.com>
> > > > >wrote:
> > > > >
> > > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > > owen.omal...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun 
> > > wrote:
> > > > >> >
> > > > >> > > and maybe a different project name?
> > > > >> > >
> > > > >> >
> > > > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > > > >> >
> > > > >> > .. Owen
> > > > >> >
> > > > >>
> > > > >> How about "Flora"?
> > > > >>
> > > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > > >>
> > > > >> -Andrew
> > > > >>
> > > >
> > > >
> > >
> >
>
>
>
> --
> - JRB
>

Re: [VOTE] Shall we release Hive 2.2.0 rc1?

2017-07-24 Thread Alan Gates 
+1, checked the signatures, did a build with a clean repo, ran RAT.

Alan.

On Fri, Jul 21, 2017 at 5:20 PM, Owen O'Malley 
wrote:

> Ok, I rolled a new RC with the Apache header problem fixed.
>
> Artifacts: https://home.apache.org/~omalley/hive-2.2.0/
> Tag: https://github.com/apache/hive/releases/tag/release-2.2.0rc1 da840b0
>
> .. Owen
>

Re: Review Request 61087: HIVE-16965 SMB join may produce incorrect results

2017-07-24 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61087/
---

(Updated July 24, 2017, 9:35 p.m.)


Review request for hive, Gopal V, Jason Dere, and Sergey Shelukhin.


Changes
---

Implemented comments from Sergey and Gopal.


Bugs: HIVE-16965
https://issues.apache.org/jira/browse/HIVE-16965


Repository: hive-git


Description
---

Usually, in a JOIN with multiple inputs (partitions), the inputs are read 
sequentially, however, incase of SMB join, the inputs are read based on key 
ordering. This invalidates the current IOContext assumption that the input path 
once set wont change unless the input changes.
This was resulting in incorrect partition information in results as it is 
derived from the input path in IOContext.
The new logic changes the input path as and when input changes.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties f66e19be3e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordSource.java 
add7d08c40 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/KeyValueInputMerger.java 
698fa7f69e 
  ql/src/test/queries/clientpositive/smb_join1.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/smb_join1.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/61087/diff/2/

Changes: https://reviews.apache.org/r/61087/diff/1-2/


Testing
---

Added a new test.


Thanks,

Deepak Jaiswal

Re: [DISCUSS] Separating out the metastore as its own TLP

2017-07-24 Thread Johndee Cloudera 
+1 Vihang, I do not really like Catalog as it could create confusion with
the Catalog daemon from impala.

On Mon, Jul 24, 2017 at 5:20 PM, Vihang Karajgaonkar 
wrote:

> Before we see a flood of name suggestions :) Why not just keep it
> Metastore? Its already well-known in the community and easy to relate to.
>
> On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates  wrote:
>
> > In the same vein Carter and Gunther suggested Omegastore.  Pick your
> > alphabet and whether it’s a catalog or a store I guess.
> >
> > Alan.
> >
> > On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin <
> ser...@hortonworks.com>
> > wrote:
> >
> > > I’d like to suggest ZCatalog.
> > >
> > > On 17/7/11, 15:41, "Lefty Leverenz"  wrote:
> > >
> > > >>> I'd like to suggest Riven.  (Owen O'Malley)
> > > >
> > > >> How about "Flora"?  (Andrew Sherman)
> > > >
> > > >Nice idea and thanks for introducing me to that book, Andrew.
> > > >
> > > >Along the same lines, how about "Honeycomb"?
> > > >
> > > >But since the idea is to make the metastore useful for many projects,
> a
> > > >generic name that starts with "Meta" would be less confusing ... even
> > > >though it breaks the tradition of Apache projects having quirky names.
> > > >Unfortunately "Metalog" is already in use.  "Metamorph" has other
> > > >connotations, but it's cool.
> > > >
> > > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting
> off
> > > >the metastore into a new project:  -0.5 for the sake of Hive and +0.5
> > for
> > > >the greater good.  Wishy-washy, that's me.
> > > >
> > > >-- Lefty
> > > >
> > > >
> > > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman <
> asher...@cloudera.com>
> > > >wrote:
> > > >
> > > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> > owen.omal...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun 
> > wrote:
> > > >> >
> > > >> > > and maybe a different project name?
> > > >> > >
> > > >> >
> > > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > > >> >
> > > >> > .. Owen
> > > >> >
> > > >>
> > > >> How about "Flora"?
> > > >>
> > > >> (Flora is the protagonist of The Bees by Laline Paull)
> > > >>
> > > >> -Andrew
> > > >>
> > >
> > >
> >
>



-- 
- JRB

Re: Review Request 61041: HIVE-17150: CREATE INDEX execute HMS out-of-transaction listener calls inside a transaction

2017-07-24 Thread Vihang Karajgaonkar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61041/#review181268
---


Ship it!




Ship It!

- Vihang Karajgaonkar


On July 21, 2017, 5:51 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61041/
> ---
> 
> (Updated July 21, 2017, 5:51 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Mohit Sabharwal, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-17150
> https://issues.apache.org/jira/browse/HIVE-17150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch adds a new parameter, HIVE_METASTORE_TRANSACTION_ACTIVE, to the 
> parameters passed to the notification listeners. This parameter has a 
> true/false value dependinf if the HMS is running in a transaction or not.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
>  6d7ee4cb824ba40537876bd0629831a19ac91d76 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/MetaStoreEventListenerConstants.java
>  a4f2d592ced6427a05177e67be27286c408744a6 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  b016920fa57501b9f07f1810b2f8010e40575efd 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  808c9c7c36fd0ba36adac7be942c4841ab0a08a8 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 58b9044930046758a83ee499692e5593cd82f9e0 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreListenerNotifier.java
>  20011ccec83e87a55de7668b86773bb817135cbd 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
> 8f6af9f346102359c3cd1a6c27000f46e1ddbae6 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 3ac4fe1604c7b0b455894b8e6293484e9226836e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 5a45051f4417352f20334319c796dc0ca2d8ad9e 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  bd33c7101f5bc01a482153f810b65b620a061636 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  94cbd5235dce0b49b92203ff92d0d2e29e8b2ca9 
> 
> 
> Diff: https://reviews.apache.org/r/61041/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: [DISCUSS] Separating out the metastore as its own TLP

2017-07-24 Thread Vihang Karajgaonkar 
Before we see a flood of name suggestions :) Why not just keep it
Metastore? Its already well-known in the community and easy to relate to.

On Mon, Jul 24, 2017 at 2:13 PM, Alan Gates  wrote:

> In the same vein Carter and Gunther suggested Omegastore.  Pick your
> alphabet and whether it’s a catalog or a store I guess.
>
> Alan.
>
> On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin 
> wrote:
>
> > I’d like to suggest ZCatalog.
> >
> > On 17/7/11, 15:41, "Lefty Leverenz"  wrote:
> >
> > >>> I'd like to suggest Riven.  (Owen O'Malley)
> > >
> > >> How about "Flora"?  (Andrew Sherman)
> > >
> > >Nice idea and thanks for introducing me to that book, Andrew.
> > >
> > >Along the same lines, how about "Honeycomb"?
> > >
> > >But since the idea is to make the metastore useful for many projects, a
> > >generic name that starts with "Meta" would be less confusing ... even
> > >though it breaks the tradition of Apache projects having quirky names.
> > >Unfortunately "Metalog" is already in use.  "Metamorph" has other
> > >connotations, but it's cool.
> > >
> > >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
> > >the metastore into a new project:  -0.5 for the sake of Hive and +0.5
> for
> > >the greater good.  Wishy-washy, that's me.
> > >
> > >-- Lefty
> > >
> > >
> > >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman 
> > >wrote:
> > >
> > >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley <
> owen.omal...@gmail.com>
> > >> wrote:
> > >>
> > >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun 
> wrote:
> > >> >
> > >> > > and maybe a different project name?
> > >> > >
> > >> >
> > >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> > >> >
> > >> > .. Owen
> > >> >
> > >>
> > >> How about "Flora"?
> > >>
> > >> (Flora is the protagonist of The Bees by Laline Paull)
> > >>
> > >> -Andrew
> > >>
> >
> >
>

Re: [DISCUSS] Separating out the metastore as its own TLP

2017-07-24 Thread Alan Gates 
In the same vein Carter and Gunther suggested Omegastore.  Pick your
alphabet and whether it’s a catalog or a store I guess.

Alan.

On Mon, Jul 24, 2017 at 1:35 PM, Sergey Shelukhin 
wrote:

> I’d like to suggest ZCatalog.
>
> On 17/7/11, 15:41, "Lefty Leverenz"  wrote:
>
> >>> I'd like to suggest Riven.  (Owen O'Malley)
> >
> >> How about "Flora"?  (Andrew Sherman)
> >
> >Nice idea and thanks for introducing me to that book, Andrew.
> >
> >Along the same lines, how about "Honeycomb"?
> >
> >But since the idea is to make the metastore useful for many projects, a
> >generic name that starts with "Meta" would be less confusing ... even
> >though it breaks the tradition of Apache projects having quirky names.
> >Unfortunately "Metalog" is already in use.  "Metamorph" has other
> >connotations, but it's cool.
> >
> >Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
> >the metastore into a new project:  -0.5 for the sake of Hive and +0.5 for
> >the greater good.  Wishy-washy, that's me.
> >
> >-- Lefty
> >
> >
> >On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman 
> >wrote:
> >
> >> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley 
> >> wrote:
> >>
> >> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun  wrote:
> >> >
> >> > > and maybe a different project name?
> >> > >
> >> >
> >> > Yes, it certainly needs a new name. I'd like to suggest Riven.
> >> >
> >> > .. Owen
> >> >
> >>
> >> How about "Flora"?
> >>
> >> (Flora is the protagonist of The Bees by Laline Paull)
> >>
> >> -Andrew
> >>
>
>

[jira] [Created] (HIVE-17160) Adding kerberos Authorization to the Druid hive integration

2017-07-24 Thread slim bouguerra (JIRA) 
slim bouguerra created HIVE-17160:
-

 Summary: Adding kerberos Authorization to the Druid hive 
integration
 Key: HIVE-17160
 URL: https://issues.apache.org/jira/browse/HIVE-17160
 Project: Hive
  Issue Type: New Feature
  Components: Druid integration
Reporter: slim bouguerra


This goal of this feature is to allow hive querying a secured druid cluster 
using kerberos credentials.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Separating out the metastore as its own TLP

2017-07-24 Thread Sergey Shelukhin 
I’d like to suggest ZCatalog.

On 17/7/11, 15:41, "Lefty Leverenz"  wrote:

>>> I'd like to suggest Riven.  (Owen O'Malley)
>
>> How about "Flora"?  (Andrew Sherman)
>
>Nice idea and thanks for introducing me to that book, Andrew.
>
>Along the same lines, how about "Honeycomb"?
>
>But since the idea is to make the metastore useful for many projects, a
>generic name that starts with "Meta" would be less confusing ... even
>though it breaks the tradition of Apache projects having quirky names.
>Unfortunately "Metalog" is already in use.  "Metamorph" has other
>connotations, but it's cool.
>
>Naming enthusiasm notwithstanding, I'm +/-0 on the idea of splitting off
>the metastore into a new project:  -0.5 for the sake of Hive and +0.5 for
>the greater good.  Wishy-washy, that's me.
>
>-- Lefty
>
>
>On Tue, Jul 11, 2017 at 1:04 PM, Andrew Sherman 
>wrote:
>
>> On Fri, Jun 30, 2017 at 5:05 PM, Owen O'Malley 
>> wrote:
>>
>> > On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun  wrote:
>> >
>> > > and maybe a different project name?
>> > >
>> >
>> > Yes, it certainly needs a new name. I'd like to suggest Riven.
>> >
>> > .. Owen
>> >
>>
>> How about "Flora"?
>>
>> (Flora is the protagonist of The Bees by Laline Paull)
>>
>> -Andrew
>>

[jira] [Created] (HIVE-17159) Make metastore a separately releasable module

2017-07-24 Thread Alan Gates (JIRA) 
Alan Gates created HIVE-17159:
-

 Summary: Make metastore a separately releasable module
 Key: HIVE-17159
 URL: https://issues.apache.org/jira/browse/HIVE-17159
 Project: Hive
  Issue Type: New Feature
  Components: Metastore
Reporter: Alan Gates
Assignee: Alan Gates


As proposed in this 
[thread|https://lists.apache.org/thread.html/5e75f45d60f0b819510814a126cfd3809dd24b1c7035a1c8c41b0c5c@%3Cdev.hive.apache.org%3E]
 on the dev list, we should move the metastore into a separately releasable 
module.  This is a POC of and potential first step towards separating out the 
metastore as a separate Apache TLP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61087: HIVE-16965 SMB join may produce incorrect results

2017-07-24 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61087/#review181246
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/KeyValueInputMerger.java
Lines 66 (patched)


IdentityHashMap - don't trust the hashCode() for KeyValueReader to be safe.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/KeyValueInputMerger.java
Line 109 (original), 130 (patched)


Clear the prev and IOContext refs - interrupts do leave leaky state behind 
sometimes


- Gopal V


On July 24, 2017, 6:47 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61087/
> ---
> 
> (Updated July 24, 2017, 6:47 p.m.)
> 
> 
> Review request for hive, Gopal V, Jason Dere, and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-16965
> https://issues.apache.org/jira/browse/HIVE-16965
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Usually, in a JOIN with multiple inputs (partitions), the inputs are read 
> sequentially, however, incase of SMB join, the inputs are read based on key 
> ordering. This invalidates the current IOContext assumption that the input 
> path once set wont change unless the input changes.
> This was resulting in incorrect partition information in results as it is 
> derived from the input path in IOContext.
> The new logic changes the input path as and when input changes.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties f66e19be3e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordSource.java 
> add7d08c40 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/KeyValueInputMerger.java 
> 698fa7f69e 
>   ql/src/test/queries/clientpositive/smb_join1.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/smb_join1.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61087/diff/1/
> 
> 
> Testing
> ---
> 
> Added a new test.
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Review Request 61087: HIVE-16965 SMB join may produce incorrect results

2017-07-24 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61087/
---

Review request for hive, Gopal V, Jason Dere, and Sergey Shelukhin.


Bugs: HIVE-16965
https://issues.apache.org/jira/browse/HIVE-16965


Repository: hive-git


Description
---

Usually, in a JOIN with multiple inputs (partitions), the inputs are read 
sequentially, however, incase of SMB join, the inputs are read based on key 
ordering. This invalidates the current IOContext assumption that the input path 
once set wont change unless the input changes.
This was resulting in incorrect partition information in results as it is 
derived from the input path in IOContext.
The new logic changes the input path as and when input changes.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties f66e19be3e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordSource.java 
add7d08c40 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/KeyValueInputMerger.java 
698fa7f69e 
  ql/src/test/queries/clientpositive/smb_join1.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/smb_join1.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/61087/diff/1/


Testing (updated)
---

Added a new test.


Thanks,

Deepak Jaiswal

Re: Review Request 61009: Extend object store to store bit vectors

2017-07-24 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/#review181239
---




common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java
Line 192 (original), 169 (patched)


Stream bos is not closed.



common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java
Lines 39 (patched)


ws



common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java
Lines 61 (patched)


Please add comments on the encoding.



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1733 (patched)


I think we shall set default to true in product only when we switch to 
CachedStore. So, lets keep false here. 
At the same time, we do want to run tests with default true. So, add this 
config with value true in data/conf/*.xml files.



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1734 (patched)


ws



metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql
Lines 1 (patched)


blob type ?



metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql
Lines 1 (patched)


varbinary type?



metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql
Lines 90 (patched)


varbinary type.



metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql
Lines 1 (patched)


blob type?



metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql
Lines 1 (patched)


varbinary type.



metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql
Lines 518 (patched)


varbinary type?



metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql
Lines 1 (patched)


bytea type.



metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql
Lines 537 (patched)


bytea type.



metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 953 (patched)


Add a comment on why index is 1 and not 0.



metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
Lines 124 (patched)


Can you also add the value: ndvEstimator.estimateNumDistinctValues() in log 
statement?



metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
Lines 189 (patched)


Please  log ndv value and column name?



metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
Lines 223 (patched)


Please log ndv value and column name.



metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
Lines 354 (patched)


Please log ndv value and column name.



metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
Lines 118 (patched)


ndv value.



metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
Lines 196 (patched)


log



metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
Lines 134 (patched)


Also add ndv value in log.



metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
Lines 152 (patched)


Log ndv value and column name.



metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
Lines 122 (patched)


Also add ndv value in log statement.



metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
Lines 139 (patched)


Log estimation column name and ndv value.



metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
Lines 122 (patched)

Re: Review Request 61009: Extend object store to store bit vectors

2017-07-24 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/
---

(Updated July 24, 2017, 6:08 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16997


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
  
common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
 e810ac5487 
  common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
d1955468a6 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java df45f2cc32 
  
common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
 PRE-CREATION 
  metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
  metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
  metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
  metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
  metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
  metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
  metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
  metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 0a70d47cca 
  metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 1cdeb6b45a 
  metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
c44dd067fc 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
a960b2d26b 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
b52c94c9fb 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
db4ec91cdb 
  metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
2dc2804343 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
3ac4fe1604 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
0e119896a5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
 d81d612e92 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
 e796df2422 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
 29a05390bf 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
 568bf0609b 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
 8eb64e0143 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
 b6b86123b2 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java
 af75bced72 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
 2da6f60167 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
 83c6c54fd2 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BinaryColumnStatsMerger.java
 af0669eb65 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/BooleanColumnStatsMerger.java
 33ff6a19f5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
 d3051a2b00 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 c013ba5c5d 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 e899bfe85f 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DecimalColumnStatsMerger.java
 4099ffcace 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DoubleColumnStatsMerger.java
 1691fc97df 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/LongColumnStatsMerger.java
 361af350fe 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/StringColumnStatsMerger.java
 8e28f907ee 
  
metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java
 2967a60fae

Re: Review Request 61010: HIVE-17128 Operation Logging leaks file descriptors as the log4j Appender is never closed

2017-07-24 Thread Andrew Sherman 


> On July 24, 2017, 9:09 a.m., Peter Vary wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
> > Line 123 (original), 124 (patched)
> > 
> >
> > I might be too pedantic, so feel free to tell me if you do not agree 
> > with me here :) Honestly!
> > 
> > It might be good to check here the closure of the 
> > LogDivertAppenderForTest too, so if something changes later we can be sure, 
> > that all of the appenders are closed.
> > 
> > If might be good to rename the test class to TestOpertionLogging, since 
> > from now on this will not only test the layout, but the appender closure 
> > too.

Thanks for the comments which I do not find pedantic. 
But in the case of the test I think I am testing the closure of the 
LogDivertAppenderForTest too (see lines 124-128). Can you take another look 
when you have time?
Thanks
-Andrew


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61010/#review181196
---


On July 22, 2017, 12:16 a.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61010/
> ---
> 
> (Updated July 22, 2017, 12:16 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Previously HIVE-16061 and HIVE-16400 changed Operation Logging to use the 
> Log4j2
> RoutingAppender to automatically output the log for each query into an
> individual operation log file.  As log4j does not know when a query is 
> finished
> it keeps the OutputStream in the subordinate Appender open even when the query
> completes.  The stream holds a file descriptor and so we leak file 
> descriptors.
> Note that we are already careful to close any streams reading from the 
> operation
> log file.  To fix this we use a technique described in the comments of
> LOG4J2-510 which uses reflection to close the subordinate Appender.  We use 
> this
> to close the per-query subordinate Appenders from both LogDivertAppender and
> LogDivertAppenderForTest.  The test in TestOperationLoggingLayout is extended 
> to
> check that the Appenders are closed when a query completes.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 
> 83f3af7440253bfbcedbc8b21d745fb71c0d7fb9 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
>  1a8337f574bb753e8c3c48a6b477b17700b05256 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java 
> e697b545984555414e27bafe92d7f22829a22687 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppenderForTest.java 
> 465844d66b92b371f457fda0d885d75fbfce6805 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 8d453d5d9153c2ec86c4adc7a68bd3b5dd249743 
> 
> 
> Diff: https://reviews.apache.org/r/61010/diff/2/
> 
> 
> Testing
> ---
> 
> Hand testing to show leak has gone.
> The test in TestOperationLoggingLayout is extended to check that the Appender 
> is closed.
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

[jira] [Created] (HIVE-17158) BeeLine Query Log and Query Result print order is not defined

2017-07-24 Thread Peter Vary (JIRA) 
Peter Vary created HIVE-17158:
-

 Summary: BeeLine Query Log and Query Result print order is not 
defined
 Key: HIVE-17158
 URL: https://issues.apache.org/jira/browse/HIVE-17158
 Project: Hive
  Issue Type: Bug
  Components: Beeline, Testing Infrastructure
Affects Versions: 3.0.0
Reporter: Peter Vary
Assignee: Peter Vary


The output of the BeeLine tests is sometimes flaky, especially if the query is 
a fast one

The output is sometime this:
{code}
PREHOOK: query: select explode(array('a', 'b'))
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
 A masked pattern was here 
POSTHOOK: query: select explode(array('a', 'b'))
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
 A masked pattern was here 
a
b
{code}

Sometime this:
{code}
a
b
PREHOOK: query: select explode(array('a', 'b'))
PREHOOK: type: QUERY
PREHOOK: Input: _dummy_database@_dummy_table
 A masked pattern was here 
POSTHOOK: query: select explode(array('a', 'b'))
POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
 A masked pattern was here 
{code}

Notice, that the actual query result is either before, or after the stuff 
printed by the hooks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61010: HIVE-17128 Operation Logging leaks file descriptors as the log4j Appender is never closed

2017-07-24 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61010/#review181196
---



Just 2 comments and they are not that important. Feel free to tell me if you 
disagree.

Even without change I think the code is ready to commit.

Thanks for the patch, and the changes!

Peter


itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
Line 123 (original), 124 (patched)


I might be too pedantic, so feel free to tell me if you do not agree with 
me here :) Honestly!

It might be good to check here the closure of the LogDivertAppenderForTest 
too, so if something changes later we can be sure, that all of the appenders 
are closed.

If might be good to rename the test class to TestOpertionLogging, since 
from now on this will not only test the layout, but the appender closure too.


- Peter Vary


On July 22, 2017, 12:16 a.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61010/
> ---
> 
> (Updated July 22, 2017, 12:16 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Previously HIVE-16061 and HIVE-16400 changed Operation Logging to use the 
> Log4j2
> RoutingAppender to automatically output the log for each query into an
> individual operation log file.  As log4j does not know when a query is 
> finished
> it keeps the OutputStream in the subordinate Appender open even when the query
> completes.  The stream holds a file descriptor and so we leak file 
> descriptors.
> Note that we are already careful to close any streams reading from the 
> operation
> log file.  To fix this we use a technique described in the comments of
> LOG4J2-510 which uses reflection to close the subordinate Appender.  We use 
> this
> to close the per-query subordinate Appenders from both LogDivertAppender and
> LogDivertAppenderForTest.  The test in TestOperationLoggingLayout is extended 
> to
> check that the Appenders are closed when a query completes.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 
> 83f3af7440253bfbcedbc8b21d745fb71c0d7fb9 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
>  1a8337f574bb753e8c3c48a6b477b17700b05256 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java 
> e697b545984555414e27bafe92d7f22829a22687 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppenderForTest.java 
> 465844d66b92b371f457fda0d885d75fbfce6805 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 8d453d5d9153c2ec86c4adc7a68bd3b5dd249743 
> 
> 
> Diff: https://reviews.apache.org/r/61010/diff/2/
> 
> 
> Testing
> ---
> 
> Hand testing to show leak has gone.
> The test in TestOperationLoggingLayout is extended to check that the Appender 
> is closed.
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>