This is an automated email from the ASF dual-hosted git repository.
jeongyoon pushed a change to branch nemo-63
in repository https://gitbox.apache.org/repos/asf/incubator-nemo.git.
from 6122ff6 address comments
add 4d3ef80 [NEMO-12] Frontend support for Scala Spark (#28)
new 565ae77 Merge branch 'master' of
https://github.com/apache/incubator-nemo into nemo-63
new f643199 nits
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
compiler/frontend/spark/pom.xml | 11 +
.../spark/core/{java => }/JavaSparkContext.java | 3 +-
.../snu/nemo/compiler/frontend/spark/core/RDD.java | 70 --
.../compiler/frontend/spark/core/SparkContext.java | 64 ++
.../spark/core/{java => }/SparkFrontendUtils.java | 111 +++-
.../frontend/spark/core/java/JavaPairRDD.java | 119 ----
.../frontend/spark/core/rdd/JavaPairRDD.java | 707 +++++++++++++++++++++
.../frontend/spark/core/{java => rdd}/JavaRDD.java | 200 +++---
.../source/SparkDatasetBoundedSourceVertex.java | 4 +-
.../source/SparkTextFileBoundedSourceVertex.java | 1 +
.../nemo/compiler/frontend/spark/sql/Dataset.java | 29 +-
.../compiler/frontend/spark/sql/SparkSession.java | 37 +-
.../frontend/spark/transform/CollectTransform.java | 13 +-
.../spark/transform/HDFSTextFileTransform.java | 5 +-
.../spark/transform/LocalTextFileTransform.java | 5 +-
.../frontend/spark/transform/MapTransform.java | 1 +
.../frontend/spark/transform/ReduceTransform.java | 2 +-
.../frontend/spark/core/rdd/PairRDDFunctions.scala | 324 ++++++++++
.../compiler/frontend/spark/core/rdd/RDD.scala | 521 +++++++++++++++
.../edu/snu/nemo/examples/spark/JavaMapReduce.java | 8 +-
.../edu/snu/nemo/examples/spark/JavaSparkPi.java | 4 +-
.../edu/snu/nemo/examples/spark/JavaWordCount.java | 4 +-
.../examples/spark/sql/JavaSparkSQLExample.java | 2 +-
.../spark/{SparkPi.scala => SparkWordCount.scala} | 45 +-
.../{SparkITCase.java => SparkJavaITCase.java} | 5 +-
.../snu/nemo/examples/spark/SparkScalaITCase.java} | 69 +-
26 files changed, 1967 insertions(+), 397 deletions(-)
rename
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/{java
=> }/JavaSparkContext.java (95%)
delete mode 100644
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/RDD.java
create mode 100644
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/SparkContext.java
rename
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/{java
=> }/SparkFrontendUtils.java (54%)
delete mode 100644
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/java/JavaPairRDD.java
create mode 100644
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/rdd/JavaPairRDD.java
rename
compiler/frontend/spark/src/main/java/edu/snu/nemo/compiler/frontend/spark/core/{java
=> rdd}/JavaRDD.java (74%)
create mode 100644
compiler/frontend/spark/src/main/scala/edu/snu/nemo/compiler/frontend/spark/core/rdd/PairRDDFunctions.scala
create mode 100644
compiler/frontend/spark/src/main/scala/edu/snu/nemo/compiler/frontend/spark/core/rdd/RDD.scala
copy examples/spark/src/main/scala/edu/snu/nemo/examples/spark/{SparkPi.scala
=> SparkWordCount.scala} (56%)
rename
examples/spark/src/test/java/edu/snu/nemo/examples/spark/{SparkITCase.java =>
SparkJavaITCase.java} (98%)
copy
examples/{beam/src/test/java/edu/snu/nemo/examples/beam/BroadcastITCase.java =>
spark/src/test/java/edu/snu/nemo/examples/spark/SparkScalaITCase.java} (53%)
--
To stop receiving notification emails like this one, please contact
jeongy...@apache.org.
