Re: Review Request 72073: HIVE-22808

2020-02-04 Thread Krisztian Kasa

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72073/
---

(Updated Feb. 5, 2020, 5:33 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Vineet Garg.


Bugs: HIVE-22808
https://issues.apache.org/jira/browse/HIVE-22808


Repository: hive-git


Description
---

HiveRelFieldTrimmer does not handle HiveTableFunctionScan


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 9bf42ed384 
  ql/src/test/results/clientpositive/except_all.q.out 020cba4287 
  ql/src/test/results/clientpositive/intersect_all_rj.q.out b8ff98ae79 
  ql/src/test/results/clientpositive/llap/intersect_all_rj.q.out cdfbc2239e 


Diff: https://reviews.apache.org/r/72073/diff/3/

Changes: https://reviews.apache.org/r/72073/diff/2-3/


Testing
---

mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestCliDriver 
-Dqfile=intersect_all_rj.q -pl itests/qtest -Pitests


Thanks,

Krisztian Kasa



[jira] [Created] (HIVE-22830) Support ALL privilege in grant option and SQL authorization

2020-02-04 Thread Deegue (Jira)
Deegue created HIVE-22830:
-

 Summary: Support ALL privilege in grant option and SQL 
authorization
 Key: HIVE-22830
 URL: https://issues.apache.org/jira/browse/HIVE-22830
 Project: Hive
  Issue Type: Improvement
Reporter: Deegue


Upgraded from Hive 1.1.0 or other 1.x version, ALL privilege should be 
supported.

When user A grant ALL on table t1 to user B, user A has ALL privilege on table 
t1, it will thrown exception like:

{code:java}
FAILED: HiveAuthzPluginException ALLUnsupported privilege type ALL
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72073: HIVE-22808

2020-02-04 Thread Vineet Garg

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72073/#review219501
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableFunctionScan.java
Line 33 (original), 34 (patched)


It is not good idea to extend Logical nodes in Hive. Fix should instead be 
in RelFieldTrimmer to handle TableFunctionScan type.
Ideally it should be fixed in calcite, but for now we can copy the 
implementation of LogicalTableFunctionScan from calcite and resuse it for 
HiveTableFunctionScan


- Vineet Garg


On Feb. 4, 2020, 5:26 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72073/
> ---
> 
> (Updated Feb. 4, 2020, 5:26 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Vineet Garg.
> 
> 
> Bugs: HIVE-22808
> https://issues.apache.org/jira/browse/HIVE-22808
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HiveRelFieldTrimmer does not handle HiveTableFunctionScan
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableFunctionScan.java
>  ffa2a1f262 
>   ql/src/test/results/clientpositive/except_all.q.out 020cba4287 
>   ql/src/test/results/clientpositive/intersect_all_rj.q.out b8ff98ae79 
>   ql/src/test/results/clientpositive/llap/intersect_all_rj.q.out cdfbc2239e 
> 
> 
> Diff: https://reviews.apache.org/r/72073/diff/2/
> 
> 
> Testing
> ---
> 
> mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestCliDriver 
> -Dqfile=intersect_all_rj.q -pl itests/qtest -Pitests
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>



[jira] [Created] (HIVE-22829) Decimal64: NVL in vectorization miss NPE with CBO on

2020-02-04 Thread Gopal Vijayaraghavan (Jira)
Gopal Vijayaraghavan created HIVE-22829:
---

 Summary: Decimal64: NVL in vectorization miss NPE with CBO on
 Key: HIVE-22829
 URL: https://issues.apache.org/jira/browse/HIVE-22829
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Reporter: Gopal Vijayaraghavan


{code}
select  
sum(NVL(ss_sales_price, 1.0BD))
from store_sales where ss_sold_date_sk %  = 1;
{code}

{code}
| notVectorizedReason: exception: 
java.lang.NullPointerException stack trace: 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4754),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4687),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeSelectOperator(Vectorizer.java:4669),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5269),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:977),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:864),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:834),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2500(Vectorizer.java:245),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2103),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2055),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:2030),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1185),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:1017),
 
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111),
 org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180), 
... |
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-22828) Decimal64: NVL & CASE statements implicitly convert decimal64 to 128

2020-02-04 Thread Gopal Vijayaraghavan (Jira)
Gopal Vijayaraghavan created HIVE-22828:
---

 Summary: Decimal64: NVL & CASE statements implicitly convert 
decimal64 to 128 
 Key: HIVE-22828
 URL: https://issues.apache.org/jira/browse/HIVE-22828
 Project: Hive
  Issue Type: Bug
Reporter: Gopal Vijayaraghavan


{code}
select  
sum(case when (ss_item_sk=1) then ss_sales_price else null end),
sum(case when (ss_item_sk=2) then ss_sales_price else ss_sales_price+1 end),
sum(case when (ss_item_sk=2) then 1.0BD+ss_sales_price else null end)
from store_sales where ss_sold_date_sk %  = 1;
{code}

{code}
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to 
org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
at 
org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:125)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal.aggregateInputSelection(VectorUDAFSumDecimal.java:113)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:221)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
{code}

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java#L3950



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Rajesh Balamohan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219494
---




ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Line 4382 (original), 4397 (patched)


Is this needed for direct insert?. In objectstores, we could have calls 
getting throttled.


- Rajesh Balamohan


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4524 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> af14e628b3 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 48e9afc496 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
> cfd7290762 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
> 

[jira] [Created] (HIVE-22827) Update Flatbuffer version

2020-02-04 Thread Jesus Camacho Rodriguez (Jira)
Jesus Camacho Rodriguez created HIVE-22827:
--

 Summary: Update Flatbuffer version
 Key: HIVE-22827
 URL: https://issues.apache.org/jira/browse/HIVE-22827
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Hive currently uses Flatbuffer 1.2.0. Other Apache projects use a more 
up-to-date version, e.g. 1.6.0.1. Upgrade to that version.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72081: HIVE-22805 Vectorization with conditional array or map is not implemented and throws an error

2020-02-04 Thread Ramesh Kumar Thangarajan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72081/#review219493
---


Ship it!




Looks good to me.

- Ramesh Kumar Thangarajan


On Feb. 4, 2020, 11:25 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72081/
> ---
> 
> (Updated Feb. 4, 2020, 11:25 a.m.)
> 
> 
> Review request for hive and Ramesh Kumar Thangarajan.
> 
> 
> Bugs: HIVE-22805
> https://issues.apache.org/jira/browse/HIVE-22805
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implemented the copySelected and shallowCopyTo methods
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/vectorization_multi_value.q PRE-CREATION 
>   ql/src/test/results/clientpositive/vectorization_multi_value.q.out 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/ListColumnVector.java
>  8cbcc029a5 
>   
> storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/MapColumnVector.java
>  3143a44ec8 
>   
> storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/MultiValuedColumnVector.java
>  028084cfc7 
> 
> 
> Diff: https://reviews.apache.org/r/72081/diff/1/
> 
> 
> Testing
> ---
> 
> query tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>



Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Marta Kuczora via Review Board


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Lines 1732-1737 (patched)
> > 
> >
> > What about using lambda here?

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7442-7443 (original), 7456-7460 (patched)
> > 
> >
> > nit: Maybe if/else

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7562-7563 (original), 7600-7604 (patched)
> > 
> >
> > nit: Maybe if/else?

Fixed it.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4524 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>   

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Marta Kuczora via Review Board


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7526-7543 (patched)
> > 
> >
> > Is this duplicated code?

Yeah, however I cannot move this whole part to a separate method, because the 
acidOp and the isDirectInsert variables both have to be set. I can create a 
separate method for getting the value of isDirectInsert and a separate method 
for getting the tmp dir.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> bb70db4524 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> af14e628b3 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Marta Kuczora via Review Board


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > Thanks for the patch! This will be very-very usefull.
> > Some minor comments, questions...

Thanks a lot for the review!!


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
> > Lines 55 (patched)
> > 
> >
> > Is this import used?

You're right, it is not used. Removed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
> > Lines 843 (patched)
> > 
> >
> > Is inheritPerms still a working stuff? I kinda remember that it was 
> > removed from Hive some time ago...

No, I think this log message was just a copy-paste error. Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Lines 1799 (patched)
> > 
> >
> > Maybe slightly different log message, so we can easily ditinguish 
> > between this and the line below

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7379 (patched)
> > 
> >
> > We might want to make this feature configurable, to turn it on/off in 
> > case we missed some edge cases

You are absolutely right. I introduced a config parameter so we can turn on/off 
this feature.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 493-494 (patched)
> > 
> >
> > nit: Formatting? Really not important, just for the completensess shake 
> > :D

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 690-691 (patched)
> > 
> >
> > nit: Formatting?

Fixed it.


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
> > Lines 1246 (patched)
> > 
> >
> > Is this table always exists? Shall we use "drop table if exists" 
> > instead?

Fixed it.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Jan. 31, 2020, 4:12 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Jan. 31, 2020, 4:12 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-04 Thread Peter Vary via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---



Thanks for the patch! This will be very-very usefull.
Some minor comments, questions...


itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
Lines 55 (patched)


Is this import used?



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Lines 843 (patched)


Is inheritPerms still a working stuff? I kinda remember that it was removed 
from Hive some time ago...



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Lines 1444 (patched)


Why is this null?



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Lines 1732-1737 (patched)


What about using lambda here?



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Lines 1799 (patched)


Maybe slightly different log message, so we can easily ditinguish between 
this and the line below



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7379 (patched)


We might want to make this feature configurable, to turn it on/off in case 
we missed some edge cases



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7442-7443 (original), 7456-7460 (patched)


nit: Maybe if/else



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7526-7543 (patched)


Is this duplicated code?



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7562-7563 (original), 7600-7604 (patched)


nit: Maybe if/else?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 493-494 (patched)


nit: Formatting? Really not important, just for the completensess shake :D



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 690-691 (patched)


nit: Formatting?



ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java
Lines 77 (patched)


We created this variable - we should use it? Maybe set it even as a 
constant?



ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Lines 1246 (patched)


Is this table always exists? Shall we use "drop table if exists" instead?


- Peter Vary


On jan. 31, 2020, 4:12 du, Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated jan. 31, 2020, 4:12 du)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
>  4c0137 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   

[jira] [Created] (HIVE-22826) ALTER TABLE RENAME COLUMN doesn't update list of bucketed column names

2020-02-04 Thread Karen Coppage (Jira)
Karen Coppage created HIVE-22826:


 Summary:  ALTER TABLE RENAME COLUMN doesn't update list of 
bucketed column names
 Key: HIVE-22826
 URL: https://issues.apache.org/jira/browse/HIVE-22826
 Project: Hive
  Issue Type: Bug
Reporter: Karen Coppage
 Attachments: unitTest.patch

Compaction for tables where a bucketed column has been renamed fails since the 
list of bucketed columns in the StorageDescriptor doesn't get updated when the 
column is renamed, therefore we can't recreate the table correctly during 
compaction.

Attached a unit test that fails.

NO PRECOMMIT TESTS



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72063: HIVE-10362: Support Type check/conversion in dynamic partition column

2020-02-04 Thread Zoltan Haindrich

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219486
---




ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 8475 (patched)


do we really need this much if-s ?

what do you think about:
* throw real exceptions - if appropriate
* introduce feature toggle for this case/check and make it enabled by 
default - so that there will be a way back to the old behaviour



ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
Line 231 (original), 231 (patched)


it's great that this will start fixing some case in which column stats was 
not there;
I guess it was not calculated correctly earlier



ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
Lines 458 (patched)


is this change expected?
is the new resultset the correct one?
do we have an agreement between spark/llap ?


- Zoltan Haindrich


On Jan. 30, 2020, 4:30 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> ---
> 
> (Updated Jan. 30, 2020, 4:30 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part 
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1' 
> from src limit 1;
> show partitions dynparttypechecknum;
> 
> Partition created will be named:
> part=1
> even though the type of `part` is int.
> 
> Solution is to cast the inserted DP columns in the SelectOperator before 
> FileSinkOperator which creates the partition dir, not after.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 5fcc367cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 
> c1aeb8f136 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb 
>   ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 
> 43bb789840 
>   ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out 
> f745b46899 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
> fc9050b2c3 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
> 95aae7286f 
>   ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab 
>   ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac 
>   ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f 
>   ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234 
>   ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a 
>   ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e 
>   ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340 
>   ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f 
>   ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18 
>   ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f 
>   ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out 
> bc6c3add54 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
> 67b62c1265 
>   
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 
> 56d5ed945b 
>   ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1 
>   ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3 
>   ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d 
> 
> 
> Diff: https://reviews.apache.org/r/72063/diff/1/
> 
> 
> Testing
> ---
> 
> There were changes in query output in two spark auto_sortmerge_join_16.q.out 
> files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>



Review Request 72081: HIVE-22805 Vectorization with conditional array or map is not implemented and throws an error

2020-02-04 Thread Peter Vary via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72081/
---

Review request for hive and Ramesh Kumar Thangarajan.


Bugs: HIVE-22805
https://issues.apache.org/jira/browse/HIVE-22805


Repository: hive-git


Description
---

Implemented the copySelected and shallowCopyTo methods


Diffs
-

  ql/src/test/queries/clientpositive/vectorization_multi_value.q PRE-CREATION 
  ql/src/test/results/clientpositive/vectorization_multi_value.q.out 
PRE-CREATION 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/ListColumnVector.java
 8cbcc029a5 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/MapColumnVector.java 
3143a44ec8 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/MultiValuedColumnVector.java
 028084cfc7 


Diff: https://reviews.apache.org/r/72081/diff/1/


Testing
---

query tests


Thanks,

Peter Vary



[jira] [Created] (HIVE-22825) Reduce directory lookup cost for acid tables

2020-02-04 Thread Rajesh Balamohan (Jira)
Rajesh Balamohan created HIVE-22825:
---

 Summary: Reduce directory lookup cost for acid tables
 Key: HIVE-22825
 URL: https://issues.apache.org/jira/browse/HIVE-22825
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Rajesh Balamohan


With objectstores, directory lookup costs are expensive. For acid tables, it 
would be good to have a directory cache to reduce number of lookup calls.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)