Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-11 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219540
---



I looked at the diff between 1:4


ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out
Line 98 (original)


it seems we are missing the stats update task here - is this expected

note: these compute_stats() things started appearing in the 1st patch 
uploaded to the reviewboard



ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
Line 454 (original), 454 (patched)


with the latest patch(#12)
we got back this null; and also the disagreement between llap/spark...

```
grep val_169 `find . -name auto_sortmerge_join_16.q.out*`|cut -d: -f1|uniq 
-c
 24 
./ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
 17 
./ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
 17 
./ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out
```


- Zoltan Haindrich


On Feb. 7, 2020, 9:40 a.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> ---
> 
> (Updated Feb. 7, 2020, 9:40 a.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part 
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1' 
> from src limit 1;
> show partitions dynparttypechecknum;
> 
> Partition created will be named:
> part=1
> even though the type of `part` is int.
> 
> Solution is to cast the inserted DP columns in the SelectOperator before 
> FileSinkOperator which creates the partition dir, not after.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590 
>   itests/src/test/resources/testconfiguration.properties 99ca9867b1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c2514eedb1 
>   ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
>   ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
> 43bb789840 
>   ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
> f745b46899 
>   ql/src/test/results/clientpositive/llap/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 
> 453d2451df 
>   ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
>   ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
>   ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
>   ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
>   ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
>   ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
>   ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
>   ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
>   ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
>   ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
>   
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
> 56d5ed945b 
>   ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
>   ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
>   ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 
> 
> 
> Diff: https://reviews.apache.org/r/72063/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-07 Thread Karen Coppage via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/
---

(Updated Feb. 7, 2020, 8:40 a.m.)


Review request for hive and Peter Vary.


Changes
---

Patch 12


Bugs: HIVE-10362
https://issues.apache.org/jira/browse/HIVE-10362


Repository: hive-git


Description
---

Example:
create table dynparttypechecknum (key int, value string) partitioned by (part 
int);
insert into dynparttypechecknum partition (part) select key, value, '1' 
from src limit 1;
show partitions dynparttypechecknum;

Partition created will be named:
part=1
even though the type of `part` is int.

Solution is to cast the inserted DP columns in the SelectOperator before 
FileSinkOperator which creates the partition dir, not after.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590 
  itests/src/test/resources/testconfiguration.properties 99ca9867b1 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2514eedb1 
  ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
  ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
  ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
43bb789840 
  ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
f745b46899 
  ql/src/test/results/clientpositive/llap/dynpart_cast.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 
453d2451df 
  ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
  ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
  ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
  ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
  ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
  ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
  ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
  ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
  ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
  ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
  ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
  ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
56d5ed945b 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 


Diff: https://reviews.apache.org/r/72063/diff/4/

Changes: https://reviews.apache.org/r/72063/diff/3-4/


Testing
---


Thanks,

Karen Coppage

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-06 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219519
---




ql/src/test/queries/clientpositive/dynpart_cast.q
Lines 2 (patched)


Thanks for addressing the comments. Just another minor comment. 
Can we add this test under 'minillap.query.files'? Right now this is MR 
test but since we use Tez/Llap it will be better under that.
You'll need to run "TestTezMiniLlapCliDriver' to genrate q file


- Vineet Garg


On Feb. 6, 2020, 2:22 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> ---
> 
> (Updated Feb. 6, 2020, 2:22 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part 
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1' 
> from src limit 1;
> show partitions dynparttypechecknum;
> 
> Partition created will be named:
> part=1
> even though the type of `part` is int.
> 
> Solution is to cast the inserted DP columns in the SelectOperator before 
> FileSinkOperator which creates the partition dir, not after.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c2514eedb1 
>   ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
>   ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
> 43bb789840 
>   ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
> f745b46899 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 
> 453d2451df 
>   ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
>   ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
>   ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
>   ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
>   ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
>   ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
>   ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
>   ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
>   ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
>   ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
>   
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
> 56d5ed945b 
>   ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
>   ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
>   ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 
> 
> 
> Diff: https://reviews.apache.org/r/72063/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-06 Thread Karen Coppage via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/
---

(Updated Feb. 6, 2020, 2:22 p.m.)


Review request for hive and Peter Vary.


Changes
---

Patch 11 - qtest changes, dropped use of DynamicPartitionCtx


Bugs: HIVE-10362
https://issues.apache.org/jira/browse/HIVE-10362


Repository: hive-git


Description
---

Example:
create table dynparttypechecknum (key int, value string) partitioned by (part 
int);
insert into dynparttypechecknum partition (part) select key, value, '1' 
from src limit 1;
show partitions dynparttypechecknum;

Partition created will be named:
part=1
even though the type of `part` is int.

Solution is to cast the inserted DP columns in the SelectOperator before 
FileSinkOperator which creates the partition dir, not after.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2514eedb1 
  ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
  ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
  ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
43bb789840 
  ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
f745b46899 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 
453d2451df 
  ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
  ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
  ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
  ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
  ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
  ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
  ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
  ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
  ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
  ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
  ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
  ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
56d5ed945b 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 


Diff: https://reviews.apache.org/r/72063/diff/3/

Changes: https://reviews.apache.org/r/72063/diff/2-3/


Testing
---

There were changes in query output in two spark auto_sortmerge_join_16.q.out 
files. They now match the query output of llap/auto_sortmerge_join_16.q.out.


Thanks,

Karen Coppage

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-05 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219513
---




ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java
Lines 88 (patched)


This will only be used if DYNAMICPARTITIONCONVERT is set to true but right 
now the map is always populated. It will be good to explicitly populate the map 
(may be by a call from genConversionSelectOperator before adding the cast)



ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java
Lines 222 (patched)


A comment would be helpful here to explain why we need to keep it



ql/src/test/queries/clientpositive/dynpart_cast.q
Lines 7 (patched)


Can we add explain plan?



ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out
Lines 36 (patched)


This test is to test Sort dynamic bucket partitioning, with the change this 
optimization is not kicking in anymore (PARTITION_BUCKET_SORTED keyword is 
missing in File sink). We should understand why is it happening and may be 
either fix it or log a jira.


- Vineet Garg


On Jan. 30, 2020, 3:30 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> ---
> 
> (Updated Jan. 30, 2020, 3:30 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part 
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1' 
> from src limit 1;
> show partitions dynparttypechecknum;
> 
> Partition created will be named:
> part=1
> even though the type of `part` is int.
> 
> Solution is to cast the inserted DP columns in the SelectOperator before 
> FileSinkOperator which creates the partition dir, not after.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c2514eedb1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 
> c1aeb8f136 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
>   ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
> 43bb789840 
>   ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
> f745b46899 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
> fc9050b2c3 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
> 95aae7286f 
>   ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab 
>   ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
>   ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
>   ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
>   ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
>   ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
>   ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
>   ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
>   ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
>   ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
>   ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out 
> bc6c3add54 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
> 67b62c1265 
>   
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
> 56d5ed945b 
>   ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
>   ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
>   ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 
> 
> 
> Diff: https://reviews.apache.org/r/72063/diff/2/
> 
> 
> Testing
> ---
> 
> There were changes in query output in two spark auto_sortmerge_join_16.q.out 
> files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
> 
> 
> Thanks,
> 
> Karen Coppage
>

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-05 Thread Karen Coppage via Review Board 


> On Feb. 4, 2020, 2:03 p.m., Zoltan Haindrich wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 8475 (patched)
> > 
> >
> > do we really need this much if-s ?
> > 
> > 
> > what do you think about:
> > * throw real exceptions - if appropriate
> > * introduce feature toggle for this case/check and make it enabled by 
> > default - so that there will be a way back to the old behaviour

Good idea, will do.


> On Feb. 4, 2020, 2:03 p.m., Zoltan Haindrich wrote:
> > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
> > Lines 458 (patched)
> > 
> >
> > is this change expected?
> > is the new resultset the correct one?
> > do we have an agreement between spark/llap ?

With this change we have an agreement with llap. I also checked the results and 
llap was correct.


- Karen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219486
---


On Jan. 30, 2020, 3:30 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> ---
> 
> (Updated Jan. 30, 2020, 3:30 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part 
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1' 
> from src limit 1;
> show partitions dynparttypechecknum;
> 
> Partition created will be named:
> part=1
> even though the type of `part` is int.
> 
> Solution is to cast the inserted DP columns in the SelectOperator before 
> FileSinkOperator which creates the partition dir, not after.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 
> c1aeb8f136 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
>   ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
> 43bb789840 
>   ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
> f745b46899 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
> fc9050b2c3 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
> 95aae7286f 
>   ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab 
>   ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
>   ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
>   ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
>   ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
>   ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
>   ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
>   ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
>   ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
>   ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
>   ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out 
> bc6c3add54 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
> 67b62c1265 
>   
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
> 56d5ed945b 
>   ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
>   ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
>   ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 
> 
> 
> Diff: https://reviews.apache.org/r/72063/diff/1/
> 
> 
> Testing
> ---
> 
> There were changes in query output in two spark auto_sortmerge_join_16.q.out 
> files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-04 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219486
---




ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 8475 (patched)


do we really need this much if-s ?

what do you think about:
* throw real exceptions - if appropriate
* introduce feature toggle for this case/check and make it enabled by 
default - so that there will be a way back to the old behaviour



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
Line 231 (original), 231 (patched)


it's great that this will start fixing some case in which column stats was 
not there;
I guess it was not calculated correctly earlier



ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
Lines 458 (patched)


is this change expected?
is the new resultset the correct one?
do we have an agreement between spark/llap ?


- Zoltan Haindrich


On Jan. 30, 2020, 4:30 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> ---
> 
> (Updated Jan. 30, 2020, 4:30 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part 
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1' 
> from src limit 1;
> show partitions dynparttypechecknum;
> 
> Partition created will be named:
> part=1
> even though the type of `part` is int.
> 
> Solution is to cast the inserted DP columns in the SelectOperator before 
> FileSinkOperator which creates the partition dir, not after.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 
> c1aeb8f136 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
>   ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
> 43bb789840 
>   ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
> f745b46899 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
> fc9050b2c3 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
> 95aae7286f 
>   ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab 
>   ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
>   ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
>   ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
>   ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
>   ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
>   ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
>   ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
>   ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
>   ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
>   ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out 
> bc6c3add54 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
> 67b62c1265 
>   
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
> 56d5ed945b 
>   ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
>   ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
>   ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 
> 
> 
> Diff: https://reviews.apache.org/r/72063/diff/1/
> 
> 
> Testing
> ---
> 
> There were changes in query output in two spark auto_sortmerge_join_16.q.out 
> files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-01-30 Thread Karen Coppage via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/
---

(Updated Jan. 30, 2020, 3:30 p.m.)


Review request for hive and Peter Vary.


Bugs: HIVE-10362
https://issues.apache.org/jira/browse/HIVE-10362


Repository: hive-git


Description
---

Example:
create table dynparttypechecknum (key int, value string) partitioned by (part 
int);
insert into dynparttypechecknum partition (part) select key, value, '1' 
from src limit 1;
show partitions dynparttypechecknum;

Partition created will be named:
part=1
even though the type of `part` is int.

Solution is to cast the inserted DP columns in the SelectOperator before 
FileSinkOperator which creates the partition dir, not after.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 
c1aeb8f136 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
2c4b69b2fe 
  ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
  ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
  ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
43bb789840 
  ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
f745b46899 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
fc9050b2c3 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
95aae7286f 
  ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab 
  ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
  ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
  ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
  ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
  ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
  ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
  ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
  ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
  ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
  ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
  ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out 
bc6c3add54 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
67b62c1265 
  ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
56d5ed945b 
  ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
  ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
  ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 


Diff: https://reviews.apache.org/r/72063/diff/1/


Testing
---

There were changes in query output in two spark auto_sortmerge_join_16.q.out 
files. They now match the query output of llap/auto_sortmerge_join_16.q.out.


Thanks,

Karen Coppage