Re: Review Request 65778: HIVE-18726 Implement DEFAULT constraint

2018-02-26 Thread Vineet Garg 


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
> > Lines 409 (patched)
> > 
> >
> > What about other types: char, varchar, timestamp, tstz, decimal etc. If 
> > those are not needed, can you add a comment about it?

UDFs for those types are already registered in FunctionRegistry (just few lines 
after this change). These were the ones missing. Will put a comment.


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/DefaultConstraint.java
> > Lines 34 (patched)
> > 
> >
> > Is there a reason to implement Serializable ?

I am not sure since I implemented this based on NotNullConstraint. I'll look 
into it.


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
> > Lines 803 (patched)
> > 
> >
> > I wonder if this is ever useful. Does standard allows currentDB?

Standard doesn't say anything about currentDB. I think I might have seen it 
somewhere else. If you don't think this is useful at all I'll get rid of it. 
Anyway it is not working for some reason.


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 4409 (patched)
> > 
> >
> > Does this mean: default value was specified as null ? We won't 
> > serialize null in metastore? If so, can you add comment about this.

No we do serialize NULL default value in metastore. This is existing code/logic 
where if you are missing column reference in INSERT statement compiler will 
create null expression e.g. INSERT INTO TABLE T1(col1, col3) values(1,3). col2 
reference is missing so NULL expression will be created. Basically if column 
reference is missing and no default value was provided NULL expression will be 
used as default.


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 4411 (patched)
> > 
> >
> > LOG.debug("Added default value from metastore: " + exp)

done


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 12595 (patched)
> > 
> >
> > They can be added later. Can you leave a TODO?

done


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
> > Lines 120-128 (patched)
> > 
> >
> > Covered in different patch.

Yes I'll remove this once I have that patch in.


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientnegative/constraint_partition_columns.q
> > Lines 1 (patched)
> > 
> >
> > We can support this later. Effectively, replace 
> > __HIVE_DEFAULT_PARTITION_ crux we have.

Yes once HIVE-18792 is done then we can support this. I have left TO-DO. I plan 
to open a JIRA once I have the patch is.


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/default_constraint.q
> > Lines 1 (patched)
> > 
> >
> > Can you create all these tables as ACID? We primarily want to test this 
> > feature for ACID tables.

Will do


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientnegative/default_constraint_invalid_default_value_type.q.out
> > Lines 1 (patched)
> > 
> >
> > Can you also add -ve test disallowing complex type in default value.

Will add one.


> On Feb. 27, 2018, 2:07 a.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 9672 (patched)
> > 
> >
> > Can you add directsql=false for one of the queries in test to cover 
> > this code path?

will do


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65778/#review198298
---


On Feb. 23, 2018, 9:52 p.m., Vineet Garg wrote:
> 
>

Re: Review Request 65415: HIVE-18571 stats issues for MM tables

2018-02-26 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65415/
---

(Updated Feb. 27, 2018, 3:14 a.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

f.,v fbghdscd


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java df77a4a2f2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java b490325091 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java fd8423129f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 0a82225d4a 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 70fcd2c142 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java 1a63d3f971 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 8b0af3e5c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
67d05e65dd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
7d2de75315 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cd6f1ee692 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java a4e770ce95 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 
8ce0cb05b6 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
946c300750 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 1d7660e8b2 
  ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java 7591c0681b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 05b0474e90 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java 
d84cf136d5 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 9c0e020351 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
 59190893e6 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 89354a2d34 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 c6e34a8a22 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
 20c10607bb 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java
 b44ff8ce47 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 50f873a013 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 2599ab103e 


Diff: https://reviews.apache.org/r/65415/diff/4/

Changes: https://reviews.apache.org/r/65415/diff/3-4/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18811) Fix desc table, column comments are not displayed

2018-02-26 Thread tartarus (JIRA) 
tartarus created HIVE-18811:
---

 Summary: Fix desc table, column comments are not displayed
 Key: HIVE-18811
 URL: https://issues.apache.org/jira/browse/HIVE-18811
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 2.3.2, 1.2.1
 Environment: CentOS 6.5

Hive-1.2.1

Hive-3.0.0
Reporter: tartarus
Assignee: tartarus
 Fix For: 3.0.0


when column comment contain \t 

eg: CREATE TABLE `zhangmang_test`(`name` string COMMENT 
'name{color:#FF}\t{color}zm');

then execute : {color:#FF}desc zhangmang_test {color}

{color:#FF}{color:#33}return : name                string              
name{color}
{color}

because \t is the separator, so we should translate it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65778: HIVE-18726 Implement DEFAULT constraint

2018-02-26 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65778/#review198298
---




ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
Lines 409 (patched)


What about other types: char, varchar, timestamp, tstz, decimal etc. If 
those are not needed, can you add a comment about it?



ql/src/java/org/apache/hadoop/hive/ql/metadata/DefaultConstraint.java
Lines 34 (patched)


Is there a reason to implement Serializable ?



ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
Lines 803 (patched)


I wonder if this is ever useful. Does standard allows currentDB?



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 4402 (patched)


can remove this.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 4409 (patched)


Does this mean: default value was specified as null ? We won't serialize 
null in metastore? If so, can you add comment about this.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 4411 (patched)


LOG.debug("Added default value from metastore: " + exp)



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 12595 (patched)


They can be added later. Can you leave a TODO?



ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
Lines 120-128 (patched)


Covered in different patch.



ql/src/test/queries/clientnegative/constraint_partition_columns.q
Lines 1 (patched)


We can support this later. Effectively, replace __HIVE_DEFAULT_PARTITION_ 
crux we have.



ql/src/test/queries/clientpositive/default_constraint.q
Lines 1 (patched)


Can you create all these tables as ACID? We primarily want to test this 
feature for ACID tables.



ql/src/test/results/clientnegative/default_constraint_invalid_default_value_type.q.out
Lines 1 (patched)


Can you also add -ve test disallowing complex type in default value.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 1538-1539 (patched)


These 2 lines Can be collapsed on single line.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 2309 (patched)


LOG.debug(queryText)



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 9672 (patched)


Can you add directsql=false for one of the queries in test to cover this 
code path?


- Ashutosh Chauhan


On Feb. 23, 2018, 9:52 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65778/
> ---
> 
> (Updated Feb. 23, 2018, 9:52 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-18726
> https://issues.apache.org/jira/browse/HIVE-18726
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds DEFAULT constraint
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  a3725c5395 
>   itests/src/test/resources/testconfiguration.properties 4a52eb5559 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 6cd7a136ae 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a8f227b775 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 84d523e1d7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java f99178dbc7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/DefaultConstraint.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 8b0af3e5c8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  77e5678f80 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
>  a5b6a4b0c3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java
>  88d5554e1d 
>   
>

[jira] [Created] (HIVE-18810) Parquet Or ORC

2018-02-26 Thread Suddhasatwa Bhaumik (JIRA) 
Suddhasatwa Bhaumik created HIVE-18810:
--

 Summary: Parquet Or ORC
 Key: HIVE-18810
 URL: https://issues.apache.org/jira/browse/HIVE-18810
 Project: Hive
  Issue Type: Test
  Components: Hive
Affects Versions: 1.1.0
 Environment: Hadoop 1.2.1

Hive 1.1
Reporter: Suddhasatwa Bhaumik


Hello Experts, 

I would like to know for which data types (based on size and complexity of 
data) should one be using Parquet or ORC tables in Hive. E.g., On Hadoop 0.20.0 
with hive 0.13, the performance of ORC tables in Hive is very good when 
accessed even by 3rd party BI systems like SAP Business Objects or Tableau; 
performing the same tests on Hadoop 1.2.1 with Hive 1.1 does not yield such 
reliability in queries, although ETL or insert/update of tables are taking 
nominal time the read performance is not within acceptable limits. 

In case of any queries, kindly advise. 

Thanks

[~suddhasatwa_bhaumik]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65801: HIVE-18797 ExprConstNodeDesc's getExprString should put appropriate qualifier with literals

2018-02-26 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65801/
---

(Updated Feb. 27, 2018, 12:38 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Support for DECIMAL, INTERVAL and DOUBLE.


Bugs: HIVE-18797
https://issues.apache.org/jira/browse/HIVE-18797


Repository: hive-git


Description
---

See JIRA


Diffs (updated)
-

  hbase-handler/src/test/results/positive/hbase_timestamp.q.out e719b087d4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java 
73f449fc28 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
ab8c22b9be 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 25efe1e1f4 
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 78c4808f5f 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 9aaa6f68a2 
  ql/src/test/results/clientpositive/cast_on_constant.q.out 7f920f41af 
  ql/src/test/results/clientpositive/constant_prop_3.q.out b754f8e7b4 
  ql/src/test/results/clientpositive/constantfolding.q.out c4fce25e41 
  ql/src/test/results/clientpositive/constprog_type.q.out 27ef1f482b 
  ql/src/test/results/clientpositive/druid_basic2.q.out 08b64d6d82 
  ql/src/test/results/clientpositive/druid_basic3.q.out ce6d0aafed 
  ql/src/test/results/clientpositive/druid_intervals.q.out c94cbe9bd4 
  ql/src/test/results/clientpositive/druid_timeseries.q.out 785cbd2114 
  ql/src/test/results/clientpositive/druid_topn.q.out 90eec3dd49 
  ql/src/test/results/clientpositive/except_all.q.out c3310f5154 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
8fea7fdf44 
  ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out 49e0d35d39 
  ql/src/test/results/clientpositive/groupby_cube1.q.out 37b8c62edd 
  ql/src/test/results/clientpositive/groupby_cube_multi_gby.q.out e6126ab2c6 
  ql/src/test/results/clientpositive/groupby_grouping_id3.q.out ac43413400 
  ql/src/test/results/clientpositive/groupby_grouping_sets1.q.out 62fc9fb550 
  ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out 43e17ec6bc 
  ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 352d4beb9f 
  ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out e368e1ee12 
  ql/src/test/results/clientpositive/groupby_grouping_sets5.q.out 1766fb9bbc 
  ql/src/test/results/clientpositive/groupby_grouping_sets6.q.out 563b110a11 
  ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
1f2cd456c8 
  ql/src/test/results/clientpositive/groupby_grouping_sets_limit.q.out 
efa1802855 
  ql/src/test/results/clientpositive/groupby_grouping_window.q.out 0f58f5182a 
  ql/src/test/results/clientpositive/groupby_rollup1.q.out a849a8d368 
  ql/src/test/results/clientpositive/groupby_rollup_empty.q.out dd53fe2140 
  ql/src/test/results/clientpositive/having.q.out 8a630751bb 
  ql/src/test/results/clientpositive/having2.q.out 67f8af8328 
  ql/src/test/results/clientpositive/implicit_cast1.q.out 6e1706a8a3 
  ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out 
72249383ee 
  ql/src/test/results/clientpositive/infer_const_type.q.out 647c910241 
  ql/src/test/results/clientpositive/interval_alt.q.out 53c1be8d05 
  ql/src/test/results/clientpositive/interval_arithmetic.q.out eba97face9 
  ql/src/test/results/clientpositive/limit_pushdown2.q.out f4cff2bcc0 
  ql/src/test/results/clientpositive/literal_ints.q.out 9a56ebd5eb 
  ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
68801f0294 
  ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
98ede4e995 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_3.q.out 
aca6b21dba 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
15a97a78c1 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
78b9a443b8 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
7b1dd40a16 
  ql/src/test/results/clientpositive/llap/except_distinct.q.out 1e5937a22d 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 43ca0d9ad7 
  ql/src/test/results/clientpositive/llap/groupby_rollup_empty.q.out 24be36ea36 
  ql/src/test/results/clientpositive/llap/having.q.out 75de96ab59 
  ql/src/test/results/clientpositive/llap/intersect_all.q.out 20a234e53b 
  ql/src/test/results/clientpositive/llap/intersect_distinct.q.out a5384eb196 
  ql/src/test/results/clientpositive/llap/intersect_merge.q.out 6f0e7f1f66 
  ql/src/test/results/clientpositive/llap/lineage2.q.out 4f59a4d59b 
  ql/src/test/results/clientpositive/llap/lineage3.q.out ca4405caf7 
  ql/src/test/results/clientpositive/llap/mrr.q.out bfa26e4ff8 
  ql/src/test/results/clientpositive/llap/multi_count_distinct_null.q.out 
66bf74f15a 
  ql/src/test/results/clientpositive/llap/orc_merge5.q.out 6eba365ff2

Re: Review Request 65801: HIVE-18797 ExprConstNodeDesc's getExprString should put appropriate qualifier with literals

2018-02-26 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65801/#review198296
---




ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
Lines 126 (patched)


Do you want to add support for decimal, interval, float. 
For Char/varchar, boolean  I don't think parser can parse them as constant 
literal.


- Ashutosh Chauhan


On Feb. 26, 2018, 1:03 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65801/
> ---
> 
> (Updated Feb. 26, 2018, 1:03 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18797
> https://issues.apache.org/jira/browse/HIVE-18797
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_timestamp.q.out e719b087d4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java 
> 73f449fc28 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> ab8c22b9be 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 25efe1e1f4 
>   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 78c4808f5f 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 9aaa6f68a2 
>   ql/src/test/results/clientpositive/cast_on_constant.q.out 7f920f41af 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out b754f8e7b4 
>   ql/src/test/results/clientpositive/constantfolding.q.out c4fce25e41 
>   ql/src/test/results/clientpositive/constprog_type.q.out 27ef1f482b 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 08b64d6d82 
>   ql/src/test/results/clientpositive/druid_basic3.q.out ce6d0aafed 
>   ql/src/test/results/clientpositive/druid_intervals.q.out c94cbe9bd4 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 785cbd2114 
>   ql/src/test/results/clientpositive/druid_topn.q.out 90eec3dd49 
>   ql/src/test/results/clientpositive/except_all.q.out c3310f5154 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
> 8fea7fdf44 
>   ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out 49e0d35d39 
>   ql/src/test/results/clientpositive/groupby_cube1.q.out 37b8c62edd 
>   ql/src/test/results/clientpositive/groupby_cube_multi_gby.q.out e6126ab2c6 
>   ql/src/test/results/clientpositive/groupby_grouping_id3.q.out ac43413400 
>   ql/src/test/results/clientpositive/groupby_grouping_sets1.q.out 62fc9fb550 
>   ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out 43e17ec6bc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 352d4beb9f 
>   ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out e368e1ee12 
>   ql/src/test/results/clientpositive/groupby_grouping_sets5.q.out 1766fb9bbc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets6.q.out 563b110a11 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 1f2cd456c8 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_limit.q.out 
> efa1802855 
>   ql/src/test/results/clientpositive/groupby_grouping_window.q.out 0f58f5182a 
>   ql/src/test/results/clientpositive/groupby_rollup1.q.out a849a8d368 
>   ql/src/test/results/clientpositive/groupby_rollup_empty.q.out dd53fe2140 
>   ql/src/test/results/clientpositive/having.q.out 8a630751bb 
>   ql/src/test/results/clientpositive/having2.q.out 67f8af8328 
>   ql/src/test/results/clientpositive/implicit_cast1.q.out 6e1706a8a3 
>   
> ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out 
> 72249383ee 
>   ql/src/test/results/clientpositive/infer_const_type.q.out 647c910241 
>   ql/src/test/results/clientpositive/interval_alt.q.out 53c1be8d05 
>   ql/src/test/results/clientpositive/interval_arithmetic.q.out eba97face9 
>   ql/src/test/results/clientpositive/limit_pushdown2.q.out f4cff2bcc0 
>   ql/src/test/results/clientpositive/literal_ints.q.out 9a56ebd5eb 
>   ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
> 68801f0294 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> 98ede4e995 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_3.q.out 
> aca6b21dba 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
> 15a97a78c1 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
> 78b9a443b8 
>   ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
> 7b1dd40a16 
>   ql/src/test/results/clientpositive/llap/except_distinct.q.out 1e5937a22d 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 43ca0d9ad7 
>

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

2018-02-26 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/
---

(Updated Feb. 26, 2018, 9:47 p.m.)


Review request for hive, Daniel Dai and Thejas Nair.


Bugs: HIVE-18264
https://issues.apache.org/jira/browse/HIVE-18264


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-18264


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 a3725c5395 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6c1a0b98cc 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 c6e34a8a22 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 7b44df4128 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f500d63725 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
 f0f650ddcf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0d132f2074 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 32ea17495f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 50f873a013 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 75ea8c4a77 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 207d842f94 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 ab6feb6f0b 
  standalone-metastore/src/test/resources/log4j2.properties 365687e1c9 


Diff: https://reviews.apache.org/r/65634/diff/3/

Changes: https://reviews.apache.org/r/65634/diff/2-3/


Testing
---


Thanks,

Vaibhav Gumashta

Re: Review Request 65801: HIVE-18797 ExprConstNodeDesc's getExprString should put appropriate qualifier with literals

2018-02-26 Thread Ashutosh Chauhan 


> On Feb. 26, 2018, 7:45 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
> > Lines 120 (patched)
> > 
> >
> > you need a space between DATE and ', else it will not parse correctly.
> 
> Vineet Garg wrote:
> Ashutosh I am able to parse this string correctly without any space. 
> According to grammar rule:
> dateLiteral
> :
> KW_DATE StringLiteral ->
> {
> 
> Parser doesn't expect any space.

I see. Cool.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65801/#review198291
---


On Feb. 26, 2018, 1:03 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65801/
> ---
> 
> (Updated Feb. 26, 2018, 1:03 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18797
> https://issues.apache.org/jira/browse/HIVE-18797
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_timestamp.q.out e719b087d4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java 
> 73f449fc28 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> ab8c22b9be 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 25efe1e1f4 
>   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 78c4808f5f 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 9aaa6f68a2 
>   ql/src/test/results/clientpositive/cast_on_constant.q.out 7f920f41af 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out b754f8e7b4 
>   ql/src/test/results/clientpositive/constantfolding.q.out c4fce25e41 
>   ql/src/test/results/clientpositive/constprog_type.q.out 27ef1f482b 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 08b64d6d82 
>   ql/src/test/results/clientpositive/druid_basic3.q.out ce6d0aafed 
>   ql/src/test/results/clientpositive/druid_intervals.q.out c94cbe9bd4 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 785cbd2114 
>   ql/src/test/results/clientpositive/druid_topn.q.out 90eec3dd49 
>   ql/src/test/results/clientpositive/except_all.q.out c3310f5154 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
> 8fea7fdf44 
>   ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out 49e0d35d39 
>   ql/src/test/results/clientpositive/groupby_cube1.q.out 37b8c62edd 
>   ql/src/test/results/clientpositive/groupby_cube_multi_gby.q.out e6126ab2c6 
>   ql/src/test/results/clientpositive/groupby_grouping_id3.q.out ac43413400 
>   ql/src/test/results/clientpositive/groupby_grouping_sets1.q.out 62fc9fb550 
>   ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out 43e17ec6bc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 352d4beb9f 
>   ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out e368e1ee12 
>   ql/src/test/results/clientpositive/groupby_grouping_sets5.q.out 1766fb9bbc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets6.q.out 563b110a11 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 1f2cd456c8 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_limit.q.out 
> efa1802855 
>   ql/src/test/results/clientpositive/groupby_grouping_window.q.out 0f58f5182a 
>   ql/src/test/results/clientpositive/groupby_rollup1.q.out a849a8d368 
>   ql/src/test/results/clientpositive/groupby_rollup_empty.q.out dd53fe2140 
>   ql/src/test/results/clientpositive/having.q.out 8a630751bb 
>   ql/src/test/results/clientpositive/having2.q.out 67f8af8328 
>   ql/src/test/results/clientpositive/implicit_cast1.q.out 6e1706a8a3 
>   
> ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out 
> 72249383ee 
>   ql/src/test/results/clientpositive/infer_const_type.q.out 647c910241 
>   ql/src/test/results/clientpositive/interval_alt.q.out 53c1be8d05 
>   ql/src/test/results/clientpositive/interval_arithmetic.q.out eba97face9 
>   ql/src/test/results/clientpositive/limit_pushdown2.q.out f4cff2bcc0 
>   ql/src/test/results/clientpositive/literal_ints.q.out 9a56ebd5eb 
>   ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
> 68801f0294 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> 98ede4e995 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_3.q.out 
> aca6b21dba 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
> 15a97a78c1 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
>

[jira] [Created] (HIVE-18809) Fix dynpart_hashjoin tests

2018-02-26 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-18809:
-

 Summary: Fix dynpart_hashjoin tests
 Key: HIVE-18809
 URL: https://issues.apache.org/jira/browse/HIVE-18809
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere


[~djaiswal] pointed out that tez_dynpart_hashjoin_2.q is not actually producing 
query plans with dynamically partitioned hash joins in the golden files. This 
test (and perhaps others) may need to be fixed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18808) Make compaction more robust when stats update fails

2018-02-26 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-18808:
-

 Summary: Make compaction more robust when stats update fails
 Key: HIVE-18808
 URL: https://issues.apache.org/jira/browse/HIVE-18808
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


 

Worker.gatherStats() runs a "analyze table..." command to update stats which 
requires SessionState.  SessionState objects are cached in ThreadLocal.  If for 
some reason Session init fails, it may still get attached to the thread which 
then causes a subsequent request that uses the same tread to gather stats fail 
because it has a bad session object.  HIVE-15658 describes the same issue in a 
different context.  

There is currently no way to recycle a session from outside HMS.

Failure to gather stats should not kill a compaction job which then prevents 
Cleaner from running.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65801: HIVE-18797 ExprConstNodeDesc's getExprString should put appropriate qualifier with literals

2018-02-26 Thread Vineet Garg 


> On Feb. 26, 2018, 7:45 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
> > Lines 120 (patched)
> > 
> >
> > you need a space between DATE and ', else it will not parse correctly.

Ashutosh I am able to parse this string correctly without any space. According 
to grammar rule:
dateLiteral
:
KW_DATE StringLiteral ->
{

Parser doesn't expect any space.


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65801/#review198291
---


On Feb. 26, 2018, 1:03 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65801/
> ---
> 
> (Updated Feb. 26, 2018, 1:03 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18797
> https://issues.apache.org/jira/browse/HIVE-18797
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_timestamp.q.out e719b087d4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java 
> 73f449fc28 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> ab8c22b9be 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 25efe1e1f4 
>   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 78c4808f5f 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 9aaa6f68a2 
>   ql/src/test/results/clientpositive/cast_on_constant.q.out 7f920f41af 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out b754f8e7b4 
>   ql/src/test/results/clientpositive/constantfolding.q.out c4fce25e41 
>   ql/src/test/results/clientpositive/constprog_type.q.out 27ef1f482b 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 08b64d6d82 
>   ql/src/test/results/clientpositive/druid_basic3.q.out ce6d0aafed 
>   ql/src/test/results/clientpositive/druid_intervals.q.out c94cbe9bd4 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 785cbd2114 
>   ql/src/test/results/clientpositive/druid_topn.q.out 90eec3dd49 
>   ql/src/test/results/clientpositive/except_all.q.out c3310f5154 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
> 8fea7fdf44 
>   ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out 49e0d35d39 
>   ql/src/test/results/clientpositive/groupby_cube1.q.out 37b8c62edd 
>   ql/src/test/results/clientpositive/groupby_cube_multi_gby.q.out e6126ab2c6 
>   ql/src/test/results/clientpositive/groupby_grouping_id3.q.out ac43413400 
>   ql/src/test/results/clientpositive/groupby_grouping_sets1.q.out 62fc9fb550 
>   ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out 43e17ec6bc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 352d4beb9f 
>   ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out e368e1ee12 
>   ql/src/test/results/clientpositive/groupby_grouping_sets5.q.out 1766fb9bbc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets6.q.out 563b110a11 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 1f2cd456c8 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_limit.q.out 
> efa1802855 
>   ql/src/test/results/clientpositive/groupby_grouping_window.q.out 0f58f5182a 
>   ql/src/test/results/clientpositive/groupby_rollup1.q.out a849a8d368 
>   ql/src/test/results/clientpositive/groupby_rollup_empty.q.out dd53fe2140 
>   ql/src/test/results/clientpositive/having.q.out 8a630751bb 
>   ql/src/test/results/clientpositive/having2.q.out 67f8af8328 
>   ql/src/test/results/clientpositive/implicit_cast1.q.out 6e1706a8a3 
>   
> ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out 
> 72249383ee 
>   ql/src/test/results/clientpositive/infer_const_type.q.out 647c910241 
>   ql/src/test/results/clientpositive/interval_alt.q.out 53c1be8d05 
>   ql/src/test/results/clientpositive/interval_arithmetic.q.out eba97face9 
>   ql/src/test/results/clientpositive/limit_pushdown2.q.out f4cff2bcc0 
>   ql/src/test/results/clientpositive/literal_ints.q.out 9a56ebd5eb 
>   ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
> 68801f0294 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> 98ede4e995 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_3.q.out 
> aca6b21dba 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
> 15a97a78c1 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
> 78b9a443b8 
>

Re: Review Request 65803: HIVE-18805 - move constantpropagate

2018-02-26 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65803/#review198293
---




ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
Lines 148 (patched)


You need to keep that becuase DPP might have introduced filters which needs 
to be removed and those will be there until 
runRemoveDynamicPruningOptimization(). So, lets keep it at both places.



ql/src/test/results/clientpositive/llap/llap_partitioned.q.out
Line 1713 (original), 1713 (patched)


New value is good. So, we are good. Value has changed for better, no need 
of investigation.



ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
Line 490 (original), 490 (patched)


ok.



ql/src/test/results/clientpositive/llap/subquery_in_having.q.out
Line 1570 (original), 1570 (patched)


No need. There are no material changes in plan/execution.



ql/src/test/results/clientpositive/llap/subquery_multi.q.out
Line 3992 (original)


cool.



ql/src/test/results/clientpositive/llap/vector_reduce_groupby_duplicate_cols.q.out
Line 62 (original), 62 (patched)


ok.


- Ashutosh Chauhan


On Feb. 26, 2018, 1:35 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65803/
> ---
> 
> (Updated Feb. 26, 2018, 1:35 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18805
> https://issues.apache.org/jira/browse/HIVE-18805
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> added an earlier invocation
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
> f9a6386ecf93f5de86de67cd383d6f140729bf08 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
> cab45c9410f44d9fdb4d06a3940be253c5ad1524 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> 43ca0d9ad718eec92663027f824f89cff9ca5de0 
>   ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 
> 5b5be1370251519e1ae236e24acf96a17c5ed311 
>   ql/src/test/results/clientpositive/llap/join46.q.out 
> 4625b9fed8fa3736c56bec91d42dfa1e7e916168 
>   ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 
> 4dd7432ac50ebef54d15f72399ecdc3d284010c4 
>   ql/src/test/results/clientpositive/llap/llap_partitioned.q.out 
> ac754674ea82c5037808d3ed2a8c9dd7fe12cca5 
>   ql/src/test/results/clientpositive/llap/mapjoin46.q.out 
> f530c1e8e1b654bc621f1619df6bbc6d09e3ae7f 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 
> cdfa73349b5913a1ab07b902095e9c1716b77cdd 
>   ql/src/test/results/clientpositive/llap/subquery_in_having.q.out 
> cc0b6a597aa2309376c62140f623debedac7aa8f 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
> 67a22c812f01a1207f6ec1d5fc8621e523f59582 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
> 50c18c8a68984dacd26cfdec9925a26e856b467d 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
> cec3daa08593e0c9eb789ef6591f8c3463224a5b 
>   ql/src/test/results/clientpositive/llap/subquery_select.q.out 
> 89eab37d07b9d37806033e1b2ebf59cbe12d014f 
>   ql/src/test/results/clientpositive/llap/tez_join_tests.q.out 
> 75879fb157397408f4bd425494f2c9d8e8426010 
>   ql/src/test/results/clientpositive/llap/tez_joins_explain.q.out 
> 3790b37760d23c9f3f49a849de6466ddff9814dc 
>   
> ql/src/test/results/clientpositive/llap/vector_reduce_groupby_duplicate_cols.q.out
>  07c1e4188c8afa9c5793c2f5890a3af9acd6a6c9 
>   ql/src/test/results/clientpositive/perf/tez/query72.q.out 
> 55ddd5a0f867052ac2cfbc1682978f8f03c1fce3 
>   ql/src/test/results/clientpositive/perf/tez/query78.q.out 
> 9c2d7b3491fa7c40ef72967e6eecca7f7f689a02 
> 
> 
> Diff: https://reviews.apache.org/r/65803/diff/1/
> 
> 
> Testing
> ---
> 
> run precommit
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>

Re: Review Request 65801: HIVE-18797 ExprConstNodeDesc's getExprString should put appropriate qualifier with literals

2018-02-26 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65801/#review198291
---




ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
Lines 120 (patched)


you need a space between DATE and ', else it will not parse correctly.



ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
Lines 122 (patched)


you need a space between DATE and ', else it will not parse correctly. Hard 
to tell if its there or not.
Similarly for TS and TSTZ.


- Ashutosh Chauhan


On Feb. 26, 2018, 1:03 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65801/
> ---
> 
> (Updated Feb. 26, 2018, 1:03 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18797
> https://issues.apache.org/jira/browse/HIVE-18797
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_timestamp.q.out e719b087d4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java 
> 73f449fc28 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> ab8c22b9be 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out 25efe1e1f4 
>   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 78c4808f5f 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 9aaa6f68a2 
>   ql/src/test/results/clientpositive/cast_on_constant.q.out 7f920f41af 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out b754f8e7b4 
>   ql/src/test/results/clientpositive/constantfolding.q.out c4fce25e41 
>   ql/src/test/results/clientpositive/constprog_type.q.out 27ef1f482b 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 08b64d6d82 
>   ql/src/test/results/clientpositive/druid_basic3.q.out ce6d0aafed 
>   ql/src/test/results/clientpositive/druid_intervals.q.out c94cbe9bd4 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 785cbd2114 
>   ql/src/test/results/clientpositive/druid_topn.q.out 90eec3dd49 
>   ql/src/test/results/clientpositive/except_all.q.out c3310f5154 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
> 8fea7fdf44 
>   ql/src/test/results/clientpositive/fold_eq_with_case_when.q.out 49e0d35d39 
>   ql/src/test/results/clientpositive/groupby_cube1.q.out 37b8c62edd 
>   ql/src/test/results/clientpositive/groupby_cube_multi_gby.q.out e6126ab2c6 
>   ql/src/test/results/clientpositive/groupby_grouping_id3.q.out ac43413400 
>   ql/src/test/results/clientpositive/groupby_grouping_sets1.q.out 62fc9fb550 
>   ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out 43e17ec6bc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 352d4beb9f 
>   ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out e368e1ee12 
>   ql/src/test/results/clientpositive/groupby_grouping_sets5.q.out 1766fb9bbc 
>   ql/src/test/results/clientpositive/groupby_grouping_sets6.q.out 563b110a11 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 1f2cd456c8 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_limit.q.out 
> efa1802855 
>   ql/src/test/results/clientpositive/groupby_grouping_window.q.out 0f58f5182a 
>   ql/src/test/results/clientpositive/groupby_rollup1.q.out a849a8d368 
>   ql/src/test/results/clientpositive/groupby_rollup_empty.q.out dd53fe2140 
>   ql/src/test/results/clientpositive/having.q.out 8a630751bb 
>   ql/src/test/results/clientpositive/having2.q.out 67f8af8328 
>   ql/src/test/results/clientpositive/implicit_cast1.q.out 6e1706a8a3 
>   
> ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out 
> 72249383ee 
>   ql/src/test/results/clientpositive/infer_const_type.q.out 647c910241 
>   ql/src/test/results/clientpositive/interval_alt.q.out 53c1be8d05 
>   ql/src/test/results/clientpositive/interval_arithmetic.q.out eba97face9 
>   ql/src/test/results/clientpositive/limit_pushdown2.q.out f4cff2bcc0 
>   ql/src/test/results/clientpositive/literal_ints.q.out 9a56ebd5eb 
>   ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
> 68801f0294 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> 98ede4e995 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_3.q.out 
> aca6b21dba 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
> 15a97a78c1 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
> 78b9a443b8 
>

[jira] [Created] (HIVE-18807) Fix broken test caused by HIVE-18493

2018-02-26 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-18807:
---

 Summary: Fix broken test caused by HIVE-18493
 Key: HIVE-18807
 URL: https://issues.apache.org/jira/browse/HIVE-18807
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18806) Add @Ignore for broken test caused by HIVE-18493

2018-02-26 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-18806:
---

 Summary: Add @Ignore for broken test caused by HIVE-18493
 Key: HIVE-18806
 URL: https://issues.apache.org/jira/browse/HIVE-18806
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65422: HIVE-17626

2018-02-26 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated Feb. 26, 2018, 6:19 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

update to 02


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
  data/conf/llap/hive-site.xml c4c299c5de 
  itests/src/test/resources/testconfiguration.properties 2776fe95f1 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae 
  ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java
 fa3f89fda1 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b 
  ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
cab45c9410 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 43ca0d9ad7 
  ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 5b5be13702 
  ql/src/test/results/clientpositive/llap/join46.q.out 4625b9fed8 
  ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 4dd7432ac5 
  ql/src/test/results/clientpositive/llap/llap_partitioned.q.out ac754674ea 
  ql/src/test/results/clientpositive/llap/mapjoin46.q.out f530c1e8e1 
  ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68 
  ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out

Re: Review Request 65803: HIVE-18805 - move constantpropagate

2018-02-26 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65803/#review198282
---



added some notes about the q.out changes


ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
Lines 148 (patched)


I will remove this FIXME on commit

should the second one be kept at the end of this function?



ql/src/test/results/clientpositive/llap/join46.q.out
Line 2089 (original)


I think that changes like this are "ok"; since it seems that the sel 
operator in this case have became a shared work



ql/src/test/results/clientpositive/llap/llap_partitioned.q.out
Line 1713 (original), 1713 (patched)


the old value seems to be totally bad...2640 aligns with the TS stat ; 
should this be investigated separetly?



ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
Line 490 (original), 490 (patched)


the datasize here is doubled; because of bug in statsutils; because I think 
the issue appears in other places as well; I would like to address this 
separetly with 
[HIVE-18804](https://issues.apache.org/jira/browse/HIVE-18804)



ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
Line 595 (original), 595 (patched)


ds multiplied; [HIVE-18804|https://issues.apache.org/jira/browse/HIVE-18804]



ql/src/test/results/clientpositive/llap/semijoin_hint.q.out
Line 744 (original), 744 (patched)


ds multiplied; [HIVE-18804|https://issues.apache.org/jira/browse/HIVE-18804]



ql/src/test/results/clientpositive/llap/subquery_in_having.q.out
Line 1570 (original), 1570 (patched)


I guess these are because some operators are not "manufactured" in the 
op-factory due to an optimization have kicked in and made it unneccessary - 
should this be investigated?



ql/src/test/results/clientpositive/llap/subquery_multi.q.out
Line 3992 (original)


this is a common filter; appearing on the other branch as well



ql/src/test/results/clientpositive/llap/vector_reduce_groupby_duplicate_cols.q.out
Line 62 (original), 62 (patched)


this is an intresting one...when stats annotation have run there was a FIL 
operator above it; which had a different stat value - might be HIVE-18804 as 
well; but around that time it was not yet clear...

anyway: the new stat value seems to fit more the plan..


- Zoltan Haindrich


On Feb. 26, 2018, 1:35 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65803/
> ---
> 
> (Updated Feb. 26, 2018, 1:35 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18805
> https://issues.apache.org/jira/browse/HIVE-18805
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> added an earlier invocation
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
> f9a6386ecf93f5de86de67cd383d6f140729bf08 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
> cab45c9410f44d9fdb4d06a3940be253c5ad1524 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> 43ca0d9ad718eec92663027f824f89cff9ca5de0 
>   ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 
> 5b5be1370251519e1ae236e24acf96a17c5ed311 
>   ql/src/test/results/clientpositive/llap/join46.q.out 
> 4625b9fed8fa3736c56bec91d42dfa1e7e916168 
>   ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 
> 4dd7432ac50ebef54d15f72399ecdc3d284010c4 
>   ql/src/test/results/clientpositive/llap/llap_partitioned.q.out 
> ac754674ea82c5037808d3ed2a8c9dd7fe12cca5 
>   ql/src/test/results/clientpositive/llap/mapjoin46.q.out 
> f530c1e8e1b654bc621f1619df6bbc6d09e3ae7f 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 
> cdfa73349b5913a1ab07b902095e9c1716b77cdd 
>   ql/src/test/results/clientpositive/llap/subquery_in_having.q.out 
> cc0b6a597aa2309376c62140f623debedac7aa8f 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
> 67a22c812f01a1207f6ec1d5fc8621e523f59582 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
> 50c18c8a68984dacd26cfdec9925a26e856b467d 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
> cec3daa08593e0c9eb789ef6591f8c3463224a5b 
>   ql/src/test/results/clientpositive/llap/subquery_select.q.out 
>

Review Request 65803: HIVE-18805 - move constantpropagate

2018-02-26 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65803/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18805
https://issues.apache.org/jira/browse/HIVE-18805


Repository: hive-git


Description
---

added an earlier invocation


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
f9a6386ecf93f5de86de67cd383d6f140729bf08 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
cab45c9410f44d9fdb4d06a3940be253c5ad1524 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
43ca0d9ad718eec92663027f824f89cff9ca5de0 
  ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 
5b5be1370251519e1ae236e24acf96a17c5ed311 
  ql/src/test/results/clientpositive/llap/join46.q.out 
4625b9fed8fa3736c56bec91d42dfa1e7e916168 
  ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 
4dd7432ac50ebef54d15f72399ecdc3d284010c4 
  ql/src/test/results/clientpositive/llap/llap_partitioned.q.out 
ac754674ea82c5037808d3ed2a8c9dd7fe12cca5 
  ql/src/test/results/clientpositive/llap/mapjoin46.q.out 
f530c1e8e1b654bc621f1619df6bbc6d09e3ae7f 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 
cdfa73349b5913a1ab07b902095e9c1716b77cdd 
  ql/src/test/results/clientpositive/llap/subquery_in_having.q.out 
cc0b6a597aa2309376c62140f623debedac7aa8f 
  ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
67a22c812f01a1207f6ec1d5fc8621e523f59582 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
50c18c8a68984dacd26cfdec9925a26e856b467d 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
cec3daa08593e0c9eb789ef6591f8c3463224a5b 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 
89eab37d07b9d37806033e1b2ebf59cbe12d014f 
  ql/src/test/results/clientpositive/llap/tez_join_tests.q.out 
75879fb157397408f4bd425494f2c9d8e8426010 
  ql/src/test/results/clientpositive/llap/tez_joins_explain.q.out 
3790b37760d23c9f3f49a849de6466ddff9814dc 
  
ql/src/test/results/clientpositive/llap/vector_reduce_groupby_duplicate_cols.q.out
 07c1e4188c8afa9c5793c2f5890a3af9acd6a6c9 
  ql/src/test/results/clientpositive/perf/tez/query72.q.out 
55ddd5a0f867052ac2cfbc1682978f8f03c1fce3 
  ql/src/test/results/clientpositive/perf/tez/query78.q.out 
9c2d7b3491fa7c40ef72967e6eecca7f7f689a02 


Diff: https://reviews.apache.org/r/65803/diff/1/


Testing
---

run precommit


Thanks,

Zoltan Haindrich

[GitHub] hive pull request #303: HIVE-18543

2018-02-26 Thread Tartarus0zm 
Github user Tartarus0zm closed the pull request at:

https://github.com/apache/hive/pull/303


---

[GitHub] hive pull request #297: HIVE-18543 Add print sessionid in console

2018-02-26 Thread Tartarus0zm 
Github user Tartarus0zm closed the pull request at:

https://github.com/apache/hive/pull/297


---

[jira] [Created] (HIVE-18805) Move ConstantPropagate before stats

2018-02-26 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18805:
---

 Summary: Move ConstantPropagate before stats
 Key: HIVE-18805
 URL: https://issues.apache.org/jira/browse/HIVE-18805
 Project: Hive
  Issue Type: Sub-task
  Components: Query Planning
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
 Attachments: HIVE-18805.01.patch

this seems to also make a few more optimizations identify more cases



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18804) StatsUtils.getColStatisticsFromExprMap may only provide info for a column once

2018-02-26 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18804:
---

 Summary: StatsUtils.getColStatisticsFromExprMap may only provide 
info for a column once
 Key: HIVE-18804
 URL: https://issues.apache.org/jira/browse/HIVE-18804
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Reporter: Zoltan Haindrich


currently {{StatsUtils.getColStatisticsFromExprMap}} may duplicate the datasize 
by passing the info about the same column more than once

https://github.com/apache/hive/blob/e8e5ab24616aa834f4966efe3a5f437f6bee4d1d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1529



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)