Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

2016-09-20 Thread cheng xu 


> On Sept. 21, 2016, 8:54 a.m., Szehon Ho wrote:
> > This looks straight-forward and good to me (once 2.0.0 is the version in 
> > pom)

Thanks Sezhon for your review. I have updated some versions required by Spark 
side.


- cheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149771
---


On Sept. 21, 2016, 1:27 p.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated Sept. 21, 2016, 1:27 p.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   itests/pom.xml a452db3 
>   pom.xml 2fb78cd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   spark-client/pom.xml 6cf3b17 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

2016-09-20 Thread cheng xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/
---

(Updated Sept. 21, 2016, 1:27 p.m.)


Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-14029
https://issues.apache.org/jira/browse/HIVE-14029


Repository: hive-git


Description
---

There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark 
to 2.0.0 to benefit those performance improvements.


Diffs (updated)
-

  itests/pom.xml a452db3 
  pom.xml 2fb78cd 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
 5b65036 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 53c5c0e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
f6595f1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java a6350d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
 09c54c1 
  ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
ee9f9b7 
  spark-client/pom.xml 6cf3b17 
  
spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java 
e77aa78 
  spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
e3b88d1 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
 e46b67d 
  spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
a7305cf 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
 be14c06 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
 4420e4d 
  
spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
 5146e91 

Diff: https://reviews.apache.org/r/48839/diff/


Testing
---


Thanks,

cheng xu

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

2016-09-20 Thread cheng xu 


> On Sept. 21, 2016, 3:44 a.m., Sahil Takiar wrote:
> > pom.xml, line 179
> > 
> >
> > Can this be changed to `2.0.0` instead of `2.0.0-preview`
> 
> Sahil Takiar wrote:
> Looked at your updated patch, seems like you already did this.

I forgot to update the review board entry. Reattach file to update it.


- cheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149717
---


On Sept. 21, 2016, 1:27 p.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated Sept. 21, 2016, 1:27 p.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   itests/pom.xml a452db3 
>   pom.xml 2fb78cd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   spark-client/pom.xml 6cf3b17 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

2016-09-20 Thread Szehon Ho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149771
---


Ship it!




This looks straight-forward and good to me (once 2.0.0 is the version in pom)

- Szehon Ho


On June 17, 2016, 8:52 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated June 17, 2016, 8:52 a.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   pom.xml 63a5ae1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

2016-09-20 Thread Sahil Takiar 


> On Sept. 20, 2016, 7:44 p.m., Sahil Takiar wrote:
> > pom.xml, line 179
> > 
> >
> > Can this be changed to `2.0.0` instead of `2.0.0-preview`

Looked at your updated patch, seems like you already did this.


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149717
---


On June 17, 2016, 8:52 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated June 17, 2016, 8:52 a.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   pom.xml 63a5ae1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

2016-09-20 Thread Sahil Takiar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149717
---




pom.xml (line 179)


Can this be changed to `2.0.0` instead of `2.0.0-preview`


- Sahil Takiar


On June 17, 2016, 8:52 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated June 17, 2016, 8:52 a.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   pom.xml 63a5ae1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

2016-06-16 Thread cheng xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/
---

Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-14029
https://issues.apache.org/jira/browse/HIVE-14029


Repository: hive-git


Description
---

There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark 
to 2.0.0 to benefit those performance improvements.


Diffs
-

  pom.xml 63a5ae1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
 5b65036 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 53c5c0e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
f6595f1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java a6350d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
 09c54c1 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
  ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
ee9f9b7 
  
spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java 
e77aa78 
  spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
e3b88d1 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
 e46b67d 
  spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
a7305cf 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
 be14c06 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
 4420e4d 
  
spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
 5146e91 

Diff: https://reviews.apache.org/r/48839/diff/


Testing
---


Thanks,

cheng xu