Re: Review Request 66514: HIVE-17645 MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)

2018-04-16 Thread Jason Dere 


> On April 16, 2018, 7:45 p.m., Sergey Shelukhin wrote:
> > llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java
> > Lines 331 (patched)
> > 
> >
> > is this related?

I threw that in, since this patch (plus this fix) also fixed 
TestAcidOnTez#testGetSplitsLocks


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66514/#review201250
---


On April 11, 2018, 7:58 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66514/
> ---
> 
> (Updated April 11, 2018, 7:58 p.m.)
> 
> 
> Review request for hive, Eugene Koifman and Sergey Shelukhin.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Replace usage of SessionState.getTxnMgr() from several places, by doing some 
> refactoring to make the TxnManager available in fields passed in during 
> construction/initialization:
> - SemanticAnalyzer.genFileSinkPlan()
> - ReplicationSemanticAnalyzer.analyzeReplLoad()
> - LoadSemanticAnalyzer.analyzeExternal()
> - ImportSemanticAnalyzer.prepareImport()
> - DDLSemanticAnalyzer.handleTransactionalTable()
> 
> 
> Diffs
> -
> 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 3aec46be51 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java bda2af3a04 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java a8d851fd81 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java 
> 6b333d7184 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadConstraint.java
>  60c85f58e5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java
>  bc7d0ad0b9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
>  06adc64727 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
>  1395027159 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/Context.java
>  bb51f36a25 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7a7bdea89d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> f38b0bc546 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 8b639f7922 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> e49089b91e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/MaterializedViewRebuildSemanticAnalyzer.java
>  e5af95b121 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
> 79b2e48ee2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 10982ddbd1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/MessageHandler.java
>  3ccd639d62 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/TableHandler.java
>  4cd75d8128 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced27e 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> fe570f0f8e 
> 
> 
> Diff: https://reviews.apache.org/r/66514/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

Re: Review Request 66514: HIVE-17645 MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)

2018-04-16 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66514/#review201250
---




llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java
Lines 331 (patched)


is this related?


- Sergey Shelukhin


On April 11, 2018, 7:58 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66514/
> ---
> 
> (Updated April 11, 2018, 7:58 p.m.)
> 
> 
> Review request for hive, Eugene Koifman and Sergey Shelukhin.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Replace usage of SessionState.getTxnMgr() from several places, by doing some 
> refactoring to make the TxnManager available in fields passed in during 
> construction/initialization:
> - SemanticAnalyzer.genFileSinkPlan()
> - ReplicationSemanticAnalyzer.analyzeReplLoad()
> - LoadSemanticAnalyzer.analyzeExternal()
> - ImportSemanticAnalyzer.prepareImport()
> - DDLSemanticAnalyzer.handleTransactionalTable()
> 
> 
> Diffs
> -
> 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 3aec46be51 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java bda2af3a04 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java a8d851fd81 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java 
> 6b333d7184 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadConstraint.java
>  60c85f58e5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java
>  bc7d0ad0b9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
>  06adc64727 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
>  1395027159 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/Context.java
>  bb51f36a25 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7a7bdea89d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> f38b0bc546 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 8b639f7922 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> e49089b91e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/MaterializedViewRebuildSemanticAnalyzer.java
>  e5af95b121 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
> 79b2e48ee2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 10982ddbd1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/MessageHandler.java
>  3ccd639d62 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/TableHandler.java
>  4cd75d8128 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced27e 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> fe570f0f8e 
> 
> 
> Diff: https://reviews.apache.org/r/66514/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

Re: Review Request 66514: HIVE-17645 MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)

2018-04-11 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66514/
---

(Updated April 11, 2018, 7:58 p.m.)


Review request for hive, Eugene Koifman and Sergey Shelukhin.


Changes
---

Added comment to SessionState.getTxnMgr() about avoiding use of this call.


Repository: hive-git


Description
---

Replace usage of SessionState.getTxnMgr() from several places, by doing some 
refactoring to make the TxnManager available in fields passed in during 
construction/initialization:
- SemanticAnalyzer.genFileSinkPlan()
- ReplicationSemanticAnalyzer.analyzeReplLoad()
- LoadSemanticAnalyzer.analyzeExternal()
- ImportSemanticAnalyzer.prepareImport()
- DDLSemanticAnalyzer.handleTransactionalTable()


Diffs (updated)
-

  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
3aec46be51 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java bda2af3a04 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java a8d851fd81 
  ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java 
6b333d7184 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadConstraint.java
 60c85f58e5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java
 bc7d0ad0b9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 06adc64727 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
 1395027159 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/Context.java
 bb51f36a25 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7a7bdea89d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
f38b0bc546 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
8b639f7922 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
e49089b91e 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/MaterializedViewRebuildSemanticAnalyzer.java
 e5af95b121 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
79b2e48ee2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 10982ddbd1 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/MessageHandler.java
 3ccd639d62 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/TableHandler.java 
4cd75d8128 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced27e 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
fe570f0f8e 


Diff: https://reviews.apache.org/r/66514/diff/3/

Changes: https://reviews.apache.org/r/66514/diff/2-3/


Testing
---


Thanks,

Jason Dere

Re: Review Request 66514: HIVE-17645 MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)

2018-04-10 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66514/
---

(Updated April 11, 2018, 1:58 a.m.)


Review request for hive, Eugene Koifman and Sergey Shelukhin.


Changes
---

Updating patch - missed a couple of uses of SessionState.getTxnMgr() from 
CalcitePlanner/MaterializedViewRebuildSemanticAnalyzer.
Also adding a couple of fixes to fix TestAcidOnTez which also depend on the 
rest of this patch.


Repository: hive-git


Description
---

Replace usage of SessionState.getTxnMgr() from several places, by doing some 
refactoring to make the TxnManager available in fields passed in during 
construction/initialization:
- SemanticAnalyzer.genFileSinkPlan()
- ReplicationSemanticAnalyzer.analyzeReplLoad()
- LoadSemanticAnalyzer.analyzeExternal()
- ImportSemanticAnalyzer.prepareImport()
- DDLSemanticAnalyzer.handleTransactionalTable()


Diffs (updated)
-

  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
3aec46be51 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java bda2af3a04 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java a8d851fd81 
  ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java 
6b333d7184 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadConstraint.java
 60c85f58e5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java
 bc7d0ad0b9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 06adc64727 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
 1395027159 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/Context.java
 bb51f36a25 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7a7bdea89d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
f38b0bc546 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
8b639f7922 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
e49089b91e 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/MaterializedViewRebuildSemanticAnalyzer.java
 e5af95b121 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
79b2e48ee2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7f0010855b 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/MessageHandler.java
 3ccd639d62 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/TableHandler.java 
4cd75d8128 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
fe570f0f8e 


Diff: https://reviews.apache.org/r/66514/diff/2/

Changes: https://reviews.apache.org/r/66514/diff/1-2/


Testing
---


Thanks,

Jason Dere

Review Request 66514: HIVE-17645 MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)

2018-04-09 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66514/
---

Review request for hive, Eugene Koifman and Sergey Shelukhin.


Repository: hive-git


Description
---

Replace usage of SessionState.getTxnMgr() from several places, by doing some 
refactoring to make the TxnManager available in fields passed in during 
construction/initialization:
- SemanticAnalyzer.genFileSinkPlan()
- ReplicationSemanticAnalyzer.analyzeReplLoad()
- LoadSemanticAnalyzer.analyzeExternal()
- ImportSemanticAnalyzer.prepareImport()
- DDLSemanticAnalyzer.handleTransactionalTable()


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java fb1efe01dc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java a8d851fd81 
  ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java 
6b333d7184 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadConstraint.java
 60c85f58e5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java
 bc7d0ad0b9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 06adc64727 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
 1395027159 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/Context.java
 bb51f36a25 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
9e66422904 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
8b639f7922 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
e49089b91e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
79b2e48ee2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ff0a2e6a1b 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/MessageHandler.java
 3ccd639d62 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/TableHandler.java 
4cd75d8128 


Diff: https://reviews.apache.org/r/66514/diff/1/


Testing
---


Thanks,

Jason Dere