Re: Review Request 64688: HIVE-18218

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/
---

(Updated Feb. 10, 2018, 5:48 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Missed the SparkOnYarn test result for auto_sortmerge_join_16.q
Fixed test bucket_mapjoin_mismatch1.q to work with new logic for file name 
format.


Repository: hive-git


Description
---

Bucket based Join : Handle buckets with no splits.

The current logic in CustomPartitionVertex assumes that there is a split for 
each bucket whereas in Tez, we can have no splits for empty buckets.
Also falls back to reduceside join if small table has more buckets than big 
table.

Disallow loading files in bucketed tables if the file name format is not like 
00_0, 01_0_copy_1 etc.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
26afe90faa 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
ef5e7edcd6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
dc698c8de8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
54f5bab6de 
  ql/src/test/queries/clientnegative/bucket_mapjoin_mismatch1.q 5f653bc9bb 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 8216b538c2 
  ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out b9c2e6f827 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
91408df129 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out 
9939e834bd 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
91408df129 


Diff: https://reviews.apache.org/r/64688/diff/5/

Changes: https://reviews.apache.org/r/64688/diff/4-5/


Testing
---


Thanks,

Deepak Jaiswal

Closing of JIRAs

[Madhudeep petwal]

Hi all,

I have noticed this that a lot of JIRAs are unnecessary, either they are
wrongly reproduced or duplicates.
One such example is this JIRA
 which I came across.
Can someone please tell me the procedure to close the JIRAs ?

As we have this large community, someone is required to clean the messes
created.
What I can do from my end is comment the name of co-ordinators and they can
review the JIRA to be closed or not.

It would be good if this procedure is provided in official docs so that
everyone can clean unnecessary JIRAs.
Please let me know your views.

Thanks
Madhudeep Petwal

Re: Review Request 64688: HIVE-18218

[Deepak Jaiswal]

> On Feb. 10, 2018, 2:44 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
> > Line 548 (original), 579 (patched)
> > 
> >
> > If a bucket file is missing in the list of files, then bucketNum < 
> > numBuckets .. so this will trigger the fallback loop below?

Yes. It is part of the fallback logic which is basically the existing logic and 
works with existing user data.

If a file is missing, the join is screwed up, just like it is right now. With 
the current naming convention it is not possible to identify a file with its 
name.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/#review197216
---


On Feb. 10, 2018, 12:41 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64688/
> ---
> 
> (Updated Feb. 10, 2018, 12:41 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket based Join : Handle buckets with no splits.
> 
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in Tez, we can have no splits for empty buckets.
> Also falls back to reduceside join if small table has more buckets than big 
> table.
> 
> Disallow loading files in bucketed tables if the file name format is not like 
> 00_0, 01_0_copy_1 etc.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
> 26afe90faa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
> ef5e7edcd6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> dc698c8de8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 54f5bab6de 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 8216b538c2 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
> 91408df129 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
> 91408df129 
> 
> 
> Diff: https://reviews.apache.org/r/64688/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 64688: HIVE-18218

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/#review197216
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java
Line 548 (original), 579 (patched)


If a bucket file is missing in the list of files, then bucketNum < 
numBuckets .. so this will trigger the fallback loop below?


- Jason Dere


On Feb. 10, 2018, 12:41 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64688/
> ---
> 
> (Updated Feb. 10, 2018, 12:41 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket based Join : Handle buckets with no splits.
> 
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in Tez, we can have no splits for empty buckets.
> Also falls back to reduceside join if small table has more buckets than big 
> table.
> 
> Disallow loading files in bucketed tables if the file name format is not like 
> 00_0, 01_0_copy_1 etc.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
> 26afe90faa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
> ef5e7edcd6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> dc698c8de8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 54f5bab6de 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 8216b538c2 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
> 91408df129 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
> 91408df129 
> 
> 
> Diff: https://reviews.apache.org/r/64688/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Review Request 65596: HIVE-18678 fix exim for MM tables and reinstante the test

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65596/
---

Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs
-

  itests/src/test/resources/testconfiguration.properties 974bfacb19 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 1f5e25f39c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 4e804ba04b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
70f5e21937 
  ql/src/test/queries/clientpositive/mm_exim.q 8a03f4d463 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/65596/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18678) fix exim for MM tables and reinstante the test

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18678:
---

 Summary: fix exim for MM tables and reinstante the test
 Key: HIVE-18678
 URL: https://issues.apache.org/jira/browse/HIVE-18678
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64688: HIVE-18218

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/
---

(Updated Feb. 10, 2018, 12:41 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Added explain plan of the query with and without SMB. The one with SMB does 
shuffle join.


Repository: hive-git


Description
---

Bucket based Join : Handle buckets with no splits.

The current logic in CustomPartitionVertex assumes that there is a split for 
each bucket whereas in Tez, we can have no splits for empty buckets.
Also falls back to reduceside join if small table has more buckets than big 
table.

Disallow loading files in bucketed tables if the file name format is not like 
00_0, 01_0_copy_1 etc.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
26afe90faa 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
ef5e7edcd6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
dc698c8de8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
54f5bab6de 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 8216b538c2 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
91408df129 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
91408df129 


Diff: https://reviews.apache.org/r/64688/diff/4/

Changes: https://reviews.apache.org/r/64688/diff/3-4/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 64688: HIVE-18218

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/
---

(Updated Feb. 10, 2018, 12:05 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

Fixed following issues,
- Handled the case when small table has more buckets than big table by taking 
mod of obtained bucket id
- Handled the fallback case for old logic when bigt table has more buckets than 
smaller table(s)
- Updated auto_sortmerge_join_16. This test would fail with SMB by default due 
to missing buckets but now gives correct results.
- Reverted all the updated tests which originally tested small tables with more 
buckets.


Repository: hive-git


Description
---

Bucket based Join : Handle buckets with no splits.

The current logic in CustomPartitionVertex assumes that there is a split for 
each bucket whereas in Tez, we can have no splits for empty buckets.
Also falls back to reduceside join if small table has more buckets than big 
table.

Disallow loading files in bucketed tables if the file name format is not like 
00_0, 01_0_copy_1 etc.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
26afe90faa 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
ef5e7edcd6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
dc698c8de8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
54f5bab6de 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 8216b538c2 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out 
91408df129 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 
91408df129 


Diff: https://reviews.apache.org/r/64688/diff/3/

Changes: https://reviews.apache.org/r/64688/diff/2-3/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-18677) SparkClientImpl usage of SessionState.LogHelper doesn't respect isSilent value

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18677:
---

 Summary: SparkClientImpl usage of SessionState.LogHelper doesn't 
respect isSilent value
 Key: HIVE-18677
 URL: https://issues.apache.org/jira/browse/HIVE-18677
 Project: Hive
  Issue Type: Sub-task
  Components: Spark, Test
Reporter: Sahil Takiar
Assignee: Sahil Takiar


In HIVE-18389 we added support for printing information to the console stream 
in the {{spark-client}} library (by passing the raw stream into 
{{SparkClientImpl}}. However, usage of the console doesn't respect the value is 
{{isSilent}} in {{SessionState}}. This has been causing a few qtests to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18676) Usability: Better warnings for Unicode single quotes

[Gopal V (JIRA)]

Gopal V created HIVE-18676:
--

 Summary: Usability: Better warnings for Unicode single quotes
 Key: HIVE-18676
 URL: https://issues.apache.org/jira/browse/HIVE-18676
 Project: Hive
  Issue Type: Bug
  Components: Parser
Reporter: Gopal V


The warning for 

{code}
select d_date as ‘date’ from date_dim;
{code}

is hard to locate than it should be.

{code}
Error: Error while compiling statement: FAILED: ParseException line 1:18 cannot 
recognize input near 'as' 'date' 'from' in selection target 
(state=42000,code=4)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-18675:
-

 Summary: make HIVE_LOCKS.HL_TXNID NOT NULL
 Key: HIVE-18675
 URL: https://issues.apache.org/jira/browse/HIVE-18675
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman


In Hive 3.0 all statements that may need locks run in a transaction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/
---

(Updated Feb. 9, 2018, 9:46 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
 5ee8aadfa7 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
 3388a34446 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
 c2728376b2 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 4e928121c7 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
 c98d22be2e 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 a5e6293a3e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 d252279be9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 68bb168bd2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 270b576199 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java abd42ec651 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 da200049bc 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
7b157e6486 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
 0fdff7d853 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
69447d9d34 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
190771ea6b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
b766791ebc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8e587f1cf6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
e1cea22005 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
0e456df19c 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
92f005d1dc 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
c6a866a164 
  
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 65508f4ddd 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
 de6c718ba9 


Diff: https://reviews.apache.org/r/65413/diff/6/

Changes: https://reviews.apache.org/r/65413/diff/5-6/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-18674) update Hive to use ORC 1.4.3

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18674:
---

 Summary: update Hive to use ORC 1.4.3
 Key: HIVE-18674
 URL: https://issues.apache.org/jira/browse/HIVE-18674
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Sergey Shelukhin]

> On Feb. 7, 2018, 7:24 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
> > Line 429 (original), 429 (patched)
> > 
> >
> > Below, you left the same compBuilder.setIsAcid(isFullAcid) but here it 
> > using isTransactional - why the logical change?
> 
> Sergey Shelukhin wrote:
> fixed; this affects whether locks are taken so my understanding is it's 
> also needed for MM
> 
> Eugene Koifman wrote:
> it doesn't affect whether locsk are taken.  locks are taken for all 
> tables even transactional=false.  it affects whether certain entries are made 
> for compactor which is not needed for MM and just pollutes those tables
> 
> Sergey Shelukhin wrote:
> I'm not sure what this has to do with compactor. This sets the flag on 
> LockComponent.
> The only place where this flag is used is 
> or (LockComponent lc : rqst.getComponent()) {
> if(lc.isSetIsAcid() && !lc.isIsAcid()) {
>   //we don't prevent using non-acid resources in a txn but we 
> do lock them
>   continue;
> }
> 
> Is it valid to treat MM table as non-acid resource?
> 
> Eugene Koifman wrote:
> isAcid here means full acid, so yes.  the locks will still be acquired

Fixed based on discussion


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196904
---


On Feb. 9, 2018, 1:52 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 9, 2018, 1:52 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc 
>   

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Eugene Koifman]

> On Feb. 7, 2018, 11:24 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
> > Line 429 (original), 429 (patched)
> > 
> >
> > Below, you left the same compBuilder.setIsAcid(isFullAcid) but here it 
> > using isTransactional - why the logical change?
> 
> Sergey Shelukhin wrote:
> fixed; this affects whether locks are taken so my understanding is it's 
> also needed for MM
> 
> Eugene Koifman wrote:
> it doesn't affect whether locsk are taken.  locks are taken for all 
> tables even transactional=false.  it affects whether certain entries are made 
> for compactor which is not needed for MM and just pollutes those tables
> 
> Sergey Shelukhin wrote:
> I'm not sure what this has to do with compactor. This sets the flag on 
> LockComponent.
> The only place where this flag is used is 
> or (LockComponent lc : rqst.getComponent()) {
> if(lc.isSetIsAcid() && !lc.isIsAcid()) {
>   //we don't prevent using non-acid resources in a txn but we 
> do lock them
>   continue;
> }
> 
> Is it valid to treat MM table as non-acid resource?

isAcid here means full acid, so yes.  the locks will still be acquired


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196904
---


On Feb. 8, 2018, 5:52 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 8, 2018, 5:52 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc 
>   

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Sergey Shelukhin]

> On Feb. 5, 2018, 6:52 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Line 507 (original), 507 (patched)
> > 
> >
> > stupid RB won't let me edit my own comment...
> > Previsouly this check was testing for full acid but now just for 
> > transactional.  Why?
> 
> Sergey Shelukhin wrote:
> It's used to skip global limit optimization. As far as I understand this 
> should also be skipped for insert-only tables
> 
> Eugene Koifman wrote:
> why is that?  For full acid, you have to apply delete events but for MM 
> once you filter the set of files based on getAcidState() it should be just 
> like non-transactional read...
> 
> Eugene Koifman wrote:
> ?

I already changed it back to full


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196819
---


On Feb. 9, 2018, 1:52 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 9, 2018, 1:52 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> c6a866a164 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  65508f4ddd 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
>  de6c718ba9 
> 
> 
> Diff: https://reviews.apache.org/r/65413/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Sergey Shelukhin]

> On Feb. 3, 2018, 1:26 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Line 507 (original), 507 (patched)
> > 
> >
> > This seems to change the logic. isAcidTable() used to mean full acid, 
> > but now it less than that...
> 
> Sergey Shelukhin wrote:
> Responded on IM :)
> 
> Eugene Koifman wrote:
> I don't see it in IM.  Why is this done?
> 
> Eugene Koifman wrote:
> I'd like an answer

This is the dup of the other comment, the change was already undone in the last 
revision


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196778
---


On Feb. 9, 2018, 1:52 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 9, 2018, 1:52 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> c6a866a164 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  65508f4ddd 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
>  de6c718ba9 
> 
> 
> Diff: https://reviews.apache.org/r/65413/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Sergey Shelukhin]

> On Feb. 7, 2018, 7:24 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
> > Line 429 (original), 429 (patched)
> > 
> >
> > Below, you left the same compBuilder.setIsAcid(isFullAcid) but here it 
> > using isTransactional - why the logical change?
> 
> Sergey Shelukhin wrote:
> fixed; this affects whether locks are taken so my understanding is it's 
> also needed for MM
> 
> Eugene Koifman wrote:
> it doesn't affect whether locsk are taken.  locks are taken for all 
> tables even transactional=false.  it affects whether certain entries are made 
> for compactor which is not needed for MM and just pollutes those tables

I'm not sure what this has to do with compactor. This sets the flag on 
LockComponent.
The only place where this flag is used is 
or (LockComponent lc : rqst.getComponent()) {
if(lc.isSetIsAcid() && !lc.isIsAcid()) {
  //we don't prevent using non-acid resources in a txn but we do 
lock them
  continue;
}

Is it valid to treat MM table as non-acid resource?


> On Feb. 7, 2018, 7:24 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
> > Line 4355 (original), 4355 (patched)
> > 
> >
> > this changes the logic.  why?
> 
> Sergey Shelukhin wrote:
> The comment just above says it's supported for ACID or MM tables :)
> 
> Eugene Koifman wrote:
> so you are making an assumption that comment is correct rather than the 
> code...

I checked with Jesus, he said it's intended to be accoring to the comment.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196904
---


On Feb. 9, 2018, 1:52 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 9, 2018, 1:52 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   

[jira] [Created] (HIVE-18673) ErrorMsg.SPARK_JOB_MONITOR_TIMEOUT isn't formatted correctly

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18673:
---

 Summary: ErrorMsg.SPARK_JOB_MONITOR_TIMEOUT isn't formatted 
correctly
 Key: HIVE-18673
 URL: https://issues.apache.org/jira/browse/HIVE-18673
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


{{ErrorMsg.SPARK_JOB_MONITOR_TIMEOUT}} doesn't format the amount of time waited 
correctly. Mainly because Java's {{MessageFormat}} class requires escaping 
single quotes with another single quote.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18672) Printed state in RemoteSparkJobMonitor is ambiguous

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18672:
---

 Summary: Printed state in RemoteSparkJobMonitor is ambiguous
 Key: HIVE-18672
 URL: https://issues.apache.org/jira/browse/HIVE-18672
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


There are a few places in {{RemoteSparkJobMonitor}} (e.g. when the Spark job is 
in state QUEUED) where the state of the Spark job is printed, but the info is 
ambiguous (no reference to HoS, or the id of the Spark job).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64688: HIVE-18218

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/#review197177
---




ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out
Line 75 (original), 56 (patched)


How does the output from this test change so much? What changed here?


- Jason Dere


On Feb. 8, 2018, 10:09 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64688/
> ---
> 
> (Updated Feb. 8, 2018, 10:09 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket based Join : Handle buckets with no splits.
> 
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in Tez, we can have no splits for empty buckets.
> Also falls back to reduceside join if small table has more buckets than big 
> table.
> 
> Disallow loading files in bucketed tables if the file name format is not like 
> 00_0, 01_0_copy_1 etc.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
> 26afe90faa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
> ef5e7edcd6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> dc698c8de8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 54f5bab6de 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 
> b9c2e6f827 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 054b0d00be 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 95d329862c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> e711715aa5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
> 53c685cb11 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 
> 8cfa113794 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 
> fce5e0cfc4 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 
> 8250eca099 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out 
> eb813c1734 
> 
> 
> Diff: https://reviews.apache.org/r/64688/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 64688: HIVE-18218

[Jason Dere]

> On Feb. 9, 2018, 5:59 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
> > Lines 582 (patched)
> > 
> >
> > Thought different number of buckets was supposed to work as long as the 
> > buckets were a multiple of each other. So this case doesn't work even if 
> > the num small buckets is a multiple of the big table buckets?

If this case is broken in the Hive-on-Tez case, can you open a followup bug for 
this issue?


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/#review197145
---


On Feb. 8, 2018, 10:09 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64688/
> ---
> 
> (Updated Feb. 8, 2018, 10:09 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket based Join : Handle buckets with no splits.
> 
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in Tez, we can have no splits for empty buckets.
> Also falls back to reduceside join if small table has more buckets than big 
> table.
> 
> Disallow loading files in bucketed tables if the file name format is not like 
> 00_0, 01_0_copy_1 etc.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
> 26afe90faa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
> ef5e7edcd6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> dc698c8de8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 54f5bab6de 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 
> b9c2e6f827 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 054b0d00be 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 95d329862c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> e711715aa5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
> 53c685cb11 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 
> 8cfa113794 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 
> fce5e0cfc4 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 
> 8250eca099 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out 
> eb813c1734 
> 
> 
> Diff: https://reviews.apache.org/r/64688/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

[jira] [Created] (HIVE-18671) lock not released after Hive on Spark query was cancelled

[Yongzhi Chen (JIRA)]

Yongzhi Chen created HIVE-18671:
---

 Summary: lock not released after Hive on Spark query was cancelled
 Key: HIVE-18671
 URL: https://issues.apache.org/jira/browse/HIVE-18671
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.2
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


When cancel the query is running on spark, the SparkJobMonitor can not return, 
therefore the locks hold by the query can not be released. When enable debug in 
log, you will see many log info as following:
{noformat}

2018-02-09 08:27:09,613 INFO 
org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: 
[HiveServer2-Background-Pool: Thread-80]: state = CANCELLED
2018-02-09 08:27:10,613 INFO 
org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: 
[HiveServer2-Background-Pool: Thread-80]: state = CANCELLED

{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64688: HIVE-18218

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64688/#review197145
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
Lines 582 (patched)


Thought different number of buckets was supposed to work as long as the 
buckets were a multiple of each other. So this case doesn't work even if the 
num small buckets is a multiple of the big table buckets?


- Jason Dere


On Feb. 8, 2018, 10:09 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64688/
> ---
> 
> (Updated Feb. 8, 2018, 10:09 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket based Join : Handle buckets with no splits.
> 
> The current logic in CustomPartitionVertex assumes that there is a split for 
> each bucket whereas in Tez, we can have no splits for empty buckets.
> Also falls back to reduceside join if small table has more buckets than big 
> table.
> 
> Disallow loading files in bucketed tables if the file name format is not like 
> 00_0, 01_0_copy_1 etc.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionVertex.java 
> 26afe90faa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomVertexConfiguration.java 
> ef5e7edcd6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 9885038588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> dc698c8de8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> 54f5bab6de 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e5fdcb57e4 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q abf09e5534 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q b85c4a7aa3 
>   ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q bd780861e3 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 
> b9c2e6f827 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 5cfc35aa73 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 0d586fd26b 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 45704d1253 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 1959075912 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 054b0d00be 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 95d329862c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> e711715aa5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
> 53c685cb11 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 
> 8cfa113794 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 
> fce5e0cfc4 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 
> 8250eca099 
>   ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out 
> eb813c1734 
> 
> 
> Diff: https://reviews.apache.org/r/64688/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Eugene Koifman]

> On Feb. 5, 2018, 10:52 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Line 507 (original), 507 (patched)
> > 
> >
> > stupid RB won't let me edit my own comment...
> > Previsouly this check was testing for full acid but now just for 
> > transactional.  Why?
> 
> Sergey Shelukhin wrote:
> It's used to skip global limit optimization. As far as I understand this 
> should also be skipped for insert-only tables
> 
> Eugene Koifman wrote:
> why is that?  For full acid, you have to apply delete events but for MM 
> once you filter the set of files based on getAcidState() it should be just 
> like non-transactional read...

?


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196819
---


On Feb. 8, 2018, 5:52 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 8, 2018, 5:52 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> c6a866a164 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  65508f4ddd 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
>  de6c718ba9 
> 
> 
> Diff: https://reviews.apache.org/r/65413/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Eugene Koifman]

> On Feb. 2, 2018, 5:26 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Line 507 (original), 507 (patched)
> > 
> >
> > This seems to change the logic. isAcidTable() used to mean full acid, 
> > but now it less than that...
> 
> Sergey Shelukhin wrote:
> Responded on IM :)
> 
> Eugene Koifman wrote:
> I don't see it in IM.  Why is this done?

I'd like an answer


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196778
---


On Feb. 8, 2018, 5:52 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 8, 2018, 5:52 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> c6a866a164 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  65508f4ddd 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
>  de6c718ba9 
> 
> 
> Diff: https://reviews.apache.org/r/65413/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables

[Eugene Koifman]

> On Feb. 7, 2018, 11:24 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
> > Line 429 (original), 429 (patched)
> > 
> >
> > Below, you left the same compBuilder.setIsAcid(isFullAcid) but here it 
> > using isTransactional - why the logical change?
> 
> Sergey Shelukhin wrote:
> fixed; this affects whether locks are taken so my understanding is it's 
> also needed for MM

it doesn't affect whether locsk are taken.  locks are taken for all tables even 
transactional=false.  it affects whether certain entries are made for compactor 
which is not needed for MM and just pollutes those tables


> On Feb. 7, 2018, 11:24 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
> > Line 4355 (original), 4355 (patched)
> > 
> >
> > this changes the logic.  why?
> 
> Sergey Shelukhin wrote:
> The comment just above says it's supported for ACID or MM tables :)

so you are making an assumption that comment is correct rather than the code...


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196904
---


On Feb. 8, 2018, 5:52 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> ---
> 
> (Updated Feb. 8, 2018, 5:52 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  5ee8aadfa7 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3388a34446 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
>  c2728376b2 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  4e928121c7 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
>  c98d22be2e 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  a5e6293a3e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  d252279be9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 
> 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  da200049bc 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> 7b157e6486 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
>  0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
> e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 92f005d1dc 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> c6a866a164 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  65508f4ddd 
>   
> 

[jira] [Created] (HIVE-18670) Prevent DROP DATABASE If Data Exists in HDFS

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18670:
--

 Summary: Prevent DROP DATABASE If Data Exists in HDFS
 Key: HIVE-18670
 URL: https://issues.apache.org/jira/browse/HIVE-18670
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 2.3.2, 3.0.0
Reporter: BELUGA BEHR


A user is not able to drop a database if it has tables under it unless they 
include the _CASCADE_ keyword to their DROP DATABASE statement.

[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DropTable]

I would like to propose that, if {{hive.mapred.mode}} is set to 'strict', Hive 
also checks for other data before dropping the database.

For example, if the database is stored within HDFS, then Hive should check if 
there exists other data, not even necessarily related to Hive, within the 
database's HDFS directory, before dropping.

The examples are:
{code:java|title=Example 1}
/hive/my_database
/hive/my_database/my_table

-- Does not succeed because 'my_table' exists
> DROP DATABASE my_database;

-- Succeeds and removes the root directory /hive/my_database
> DROP DATABASE my_database CASCADE;
{code}
{code:java|title=Example 2}
/hive/my_database
/hive/my_database/my_important_file.txt

-- Succeeds because no tables exist, but I just lost my "important" file
> DROP DATABASE my_database;
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18669) Deprecate hive.mapred.mode For New Name

[BELUGA BEHR (JIRA)]

BELUGA BEHR created HIVE-18669:
--

 Summary: Deprecate hive.mapred.mode For New Name
 Key: HIVE-18669
 URL: https://issues.apache.org/jira/browse/HIVE-18669
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 2.3.2, 3.0.0
Reporter: BELUGA BEHR


The configuration {{hive.mapred.mode}} should be changed to reflect the fact 
that 'mapred' isn't the only game in town anymore when we're talking about 
"Hive operations."

https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

{quote}
The mode in which the Hive operations are being performed. In strict mode, some 
risky queries are not allowed to run. For example, full table scans are 
prevented (see HIVE-10454) and ORDER BY requires a LIMIT clause.
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

[cheng xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/
---

(Updated Feb. 9, 2018, 8:07 p.m.)


Review request for hive.


Repository: hive-git


Description
---

VectorizedParquetReader throws an exception when trying to reading from a 
parquet table on which new columns are added.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
 907a9b8 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReader.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java
 c36640d 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 08ac57b 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java
 39689f1 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
 9e414dc 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedDictionaryEncodingColumnReader.java
 3e5d831 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
 5d3ebd6 
  
ql/src/test/queries/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q
 PRE-CREATION 
  
ql/src/test/queries/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q
 PRE-CREATION 
  ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
PRE-CREATION 
  
ql/src/test/results/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q.out
 PRE-CREATION 
  
ql/src/test/results/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q.out
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65478/diff/3/

Changes: https://reviews.apache.org/r/65478/diff/2-3/


Testing
---

Newly added UT passed and qtest passed locally.


Thanks,

cheng xu

[jira] [Created] (HIVE-18668) Really shade guava in ql

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-18668:
---

 Summary: Really shade guava in ql
 Key: HIVE-18668
 URL: https://issues.apache.org/jira/browse/HIVE-18668
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


After HIVE-15393 a test started to fail in druid; after some investigation it 
turned out that ql doesn't shade it's guava artifact at all...because it shades 
'com.google.guava' instead 'com.google.common'






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)