git commit: [SPARK-1275] Made dev/run-tests executable.
Repository: spark Updated Branches: refs/heads/branch-0.9 72875b29b - a4eef655c [SPARK-1275] Made dev/run-tests executable. This was causing Jenkins tests to fail for PRs against branch 0.9. Author: Tathagata Das tathagata.das1...@gmail.com Closes #178 from tdas/branch-0.9-fix and squashes the following commits: a633bce [Tathagata Das] Merge remote-tracking branch 'apache-github/branch-0.9' into branch-0.9-fix 9b043cc [Tathagata Das] Made dev/run-tests executable. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a4eef655 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a4eef655 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a4eef655 Branch: refs/heads/branch-0.9 Commit: a4eef655c7263f73db6f2c837288982140f17a23 Parents: 72875b2 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Wed Mar 19 16:10:45 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Wed Mar 19 16:10:45 2014 -0700 -- dev/run-tests | 0 1 file changed, 0 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/a4eef655/dev/run-tests -- diff --git a/dev/run-tests b/dev/run-tests old mode 100644 new mode 100755
git commit: Bumped versions to Spark 0.9.1
Repository: spark
Updated Branches:
refs/heads/branch-0.9 c6630d363 - 8a882eff9
Bumped versions to Spark 0.9.1
Self explanatory!
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #190 from tdas/branch-0.9-version-bump and squashes the following
commits:
34576ee [Tathagata Das] Bumped versions to Spark 0.9.1
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8a882eff
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8a882eff
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8a882eff
Branch: refs/heads/branch-0.9
Commit: 8a882eff9d4a7377ba2f470a59a3ab4722c66d9d
Parents: c6630d3
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu Mar 20 16:55:35 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Mar 20 16:55:35 2014 -0700
--
docs/_config.yml | 4 ++--
ec2/spark_ec2.py | 4 ++--
pom.xml| 2 +-
project/SparkBuild.scala | 2 +-
python/pyspark/shell.py| 2 +-
repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala | 2 +-
6 files changed, 8 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/8a882eff/docs/_config.yml
--
diff --git a/docs/_config.yml b/docs/_config.yml
index a7f46ff..db6cf74 100644
--- a/docs/_config.yml
+++ b/docs/_config.yml
@@ -3,8 +3,8 @@ markdown: kramdown
# These allow the documentation to be updated with nerw releases
# of Spark, Scala, and Mesos.
-SPARK_VERSION: 0.9.0-incubating
-SPARK_VERSION_SHORT: 0.9.0
+SPARK_VERSION: 0.9.1
+SPARK_VERSION_SHORT: 0.9.1
SCALA_BINARY_VERSION: 2.10
SCALA_VERSION: 2.10.3
MESOS_VERSION: 0.13.0
http://git-wip-us.apache.org/repos/asf/spark/blob/8a882eff/ec2/spark_ec2.py
--
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index e88f80a..ca944ce 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -70,7 +70,7 @@ def parse_args():
slaves across multiple (an additional $0.01/Gb for bandwidth +
between zones applies))
parser.add_option(-a, --ami, help=Amazon Machine Image ID to use)
- parser.add_option(-v, --spark-version, default=0.9.0,
+ parser.add_option(-v, --spark-version, default=0.9.1,
help=Version of Spark to use: 'X.Y.Z' or a specific git hash)
parser.add_option(--spark-git-repo,
default=https://github.com/apache/incubator-spark;,
@@ -157,7 +157,7 @@ def is_active(instance):
# Return correct versions of Spark and Shark, given the supplied Spark version
def get_spark_shark_version(opts):
- spark_shark_map = {0.7.3: 0.7.1, 0.8.0: 0.8.0, 0.8.1: 0.8.1,
0.9.0: 0.9.0}
+ spark_shark_map = {0.7.3: 0.7.1, 0.8.0: 0.8.0, 0.8.1: 0.8.1,
0.9.0: 0.9.0, 0.9.1: 0.9.1}
version = opts.spark_version.replace(v, )
if version not in spark_shark_map:
print stderr, Don't know about Spark version: %s % version
http://git-wip-us.apache.org/repos/asf/spark/blob/8a882eff/pom.xml
--
diff --git a/pom.xml b/pom.xml
index e53c930..f7ee914 100644
--- a/pom.xml
+++ b/pom.xml
@@ -25,7 +25,7 @@
/parent
groupIdorg.apache.spark/groupId
artifactIdspark-parent/artifactId
- version0.9.1-incubating-SNAPSHOT/version
+ version0.9.1-SNAPSHOT/version
packagingpom/packaging
nameSpark Project Parent POM/name
urlhttp://spark.incubator.apache.org//url
http://git-wip-us.apache.org/repos/asf/spark/blob/8a882eff/project/SparkBuild.scala
--
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index f9eeeb0..c5d6327 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -134,7 +134,7 @@ object SparkBuild extends Build {
def sharedSettings = Defaults.defaultSettings ++ Seq(
organization := org.apache.spark,
-version:= 0.9.0-incubating,
+version:= 0.9.1,
scalaVersion := 2.10.3,
scalacOptions := Seq(-Xmax-classfile-name, 120, -unchecked,
-deprecation,
-target: + SCALAC_JVM_VERSION),
http://git-wip-us.apache.org/repos/asf/spark/blob/8a882eff/python/pyspark/shell.py
--
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index 9203342..0425192 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -35,7 +35,7 @@ print Welcome
[1/3] git commit: Revert [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-0.9 49016046d - 348f54b9c Revert [maven-release-plugin] prepare for next development iteration This reverts commit 1f785d46e4e3df716dc836e38933dc0a30028496. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d16e863d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d16e863d Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d16e863d Branch: refs/heads/branch-0.9 Commit: d16e863de400d47bbf2b6dc6ee8d9d67db7cd02a Parents: 4901604 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Wed Mar 26 21:51:40 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Wed Mar 26 21:51:40 2014 -0700 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 18 files changed, 19 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/d16e863d/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 0edbdd6..eb884b3 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.2-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d16e863d/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index b4b5588..4ca574d 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.2-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d16e863d/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 0fdf380..8111400 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.2-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d16e863d/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 831a2df..9ae69e9 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.2-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d16e863d/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index fcd19e7..655fed1 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.2-SNAPSHOT/version +version0.9.1/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d16e863d/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index e495877..0814d33 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.2-SNAPSHOT/version +version0.9.1/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d16e863d/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index 33e3077..7a67e14 100644 --- a/external/mqtt/pom.xml +++
[2/3] git commit: Revert [maven-release-plugin] prepare release v0.9.1-rc2
Revert [maven-release-plugin] prepare release v0.9.1-rc2 This reverts commit 1197280acf1322165301259dd825f44e22a323bc. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ea5da04f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ea5da04f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ea5da04f Branch: refs/heads/branch-0.9 Commit: ea5da04f77b1de55a1d3ea03afef5dbc8c298894 Parents: d16e863 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Wed Mar 26 21:53:07 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Wed Mar 26 21:53:07 2014 -0700 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 18 files changed, 19 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/ea5da04f/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index eb884b3..d22ffc8 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1/version +version0.9.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ea5da04f/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 4ca574d..4a86e1b6 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1/version +version0.9.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ea5da04f/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 8111400..aea662a 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1/version +version0.9.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ea5da04f/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 9ae69e9..568f55e 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1/version +version0.9.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ea5da04f/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 655fed1..8896477 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1/version +version0.9.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ea5da04f/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 0814d33..8d5ebb5 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1/version +version0.9.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ea5da04f/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index 7a67e14..61092e5 100644 --- a/external/mqtt/pom.xml +++ b/external/mqtt/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId
git commit: [maven-release-plugin] prepare release v0.9.1-rc3
Repository: spark Updated Branches: refs/heads/branch-0.9 348f54b9c - 4c43182b6 [maven-release-plugin] prepare release v0.9.1-rc3 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4c43182b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4c43182b Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4c43182b Branch: refs/heads/branch-0.9 Commit: 4c43182b6d1b0b7717423f386c0214fe93073208 Parents: 348f54b Author: Tathagata Das tathagata.das1...@gmail.com Authored: Thu Mar 27 05:14:46 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Thu Mar 27 05:14:46 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 18 files changed, 19 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/4c43182b/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index d22ffc8..eb884b3 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4c43182b/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 4a86e1b6..4ca574d 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4c43182b/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index aea662a..8111400 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4c43182b/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 568f55e..9ae69e9 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1-SNAPSHOT/version +version0.9.1/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4c43182b/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 8896477..655fed1 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1-SNAPSHOT/version +version0.9.1/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4c43182b/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 8d5ebb5..0814d33 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version0.9.1-SNAPSHOT/version +version0.9.1/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4c43182b/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index 61092e5..7a67e14 100644 --- a/external/mqtt/pom.xml +++ b/external/mqtt/pom.xml @@ -21,7 +21,7 @@ parent
svn commit: r1585872 - in /spark/site/docs/0.9.1: ./ api/ api/bagel/ api/bagel/index/ api/bagel/lib/ api/bagel/org/ api/bagel/org/apache/ api/bagel/org/apache/spark/ api/bagel/org/apache/spark/bagel/
Author: tdas Date: Wed Apr 9 01:34:23 2014 New Revision: 1585872 URL: http://svn.apache.org/r1585872 Log: Added 0.9.1 docs [This commit notification would consist of 309 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]
Git Push Summary
Repository: spark Updated Tags: refs/tags/v0.9.1 [created] 4c43182b6
svn commit: r5003 - /dev/spark/spark-0.9.1/
Author: tdas Date: Wed Apr 9 19:48:15 2014 New Revision: 5003 Log: Moved Spark 0.9.1 from dev tree to release tree, within incubator/spark as a temporary place until www.apache.org/dist/spark is created. Removed: dev/spark/spark-0.9.1/
svn commit: r1586127 [2/2] - in /spark: ./ news/_posts/ releases/_posts/ site/ site/docs/ site/mllib/ site/news/ site/releases/ site/screencasts/ site/streaming/
Modified: spark/site/releases/spark-release-0-5-0.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-0-5-0.html?rev=1586127r1=1586126r2=1586127view=diff == --- spark/site/releases/spark-release-0-5-0.html (original) +++ spark/site/releases/spark-release-0-5-0.html Wed Apr 9 20:17:50 2014 @@ -124,6 +124,9 @@ h5Latest News/h5 ul class=list-unstyled + lia href=/news/spark-0-9-1-released.htmlSpark 0.9.1 released/a + span class=small(Apr 09, 2014)/span/li + lia href=/news/submit-talks-to-spark-summit-2014.htmlSubmissions and registration open for Spark Summit 2014/a span class=small(Mar 20, 2014)/span/li @@ -133,9 +136,6 @@ lia href=/news/spark-0-9-0-released.htmlSpark 0.9.0 released/a span class=small(Feb 02, 2014)/span/li - lia href=/news/spark-0-8-1-released.htmlSpark 0.8.1 released/a - span class=small(Dec 19, 2013)/span/li - /ul p class=small style=text-align: right;a href=/news/index.htmlArchive/a/p /div Modified: spark/site/releases/spark-release-0-5-1.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-0-5-1.html?rev=1586127r1=1586126r2=1586127view=diff == --- spark/site/releases/spark-release-0-5-1.html (original) +++ spark/site/releases/spark-release-0-5-1.html Wed Apr 9 20:17:50 2014 @@ -124,6 +124,9 @@ h5Latest News/h5 ul class=list-unstyled + lia href=/news/spark-0-9-1-released.htmlSpark 0.9.1 released/a + span class=small(Apr 09, 2014)/span/li + lia href=/news/submit-talks-to-spark-summit-2014.htmlSubmissions and registration open for Spark Summit 2014/a span class=small(Mar 20, 2014)/span/li @@ -133,9 +136,6 @@ lia href=/news/spark-0-9-0-released.htmlSpark 0.9.0 released/a span class=small(Feb 02, 2014)/span/li - lia href=/news/spark-0-8-1-released.htmlSpark 0.8.1 released/a - span class=small(Dec 19, 2013)/span/li - /ul p class=small style=text-align: right;a href=/news/index.htmlArchive/a/p /div Modified: spark/site/releases/spark-release-0-5-2.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-0-5-2.html?rev=1586127r1=1586126r2=1586127view=diff == --- spark/site/releases/spark-release-0-5-2.html (original) +++ spark/site/releases/spark-release-0-5-2.html Wed Apr 9 20:17:50 2014 @@ -124,6 +124,9 @@ h5Latest News/h5 ul class=list-unstyled + lia href=/news/spark-0-9-1-released.htmlSpark 0.9.1 released/a + span class=small(Apr 09, 2014)/span/li + lia href=/news/submit-talks-to-spark-summit-2014.htmlSubmissions and registration open for Spark Summit 2014/a span class=small(Mar 20, 2014)/span/li @@ -133,9 +136,6 @@ lia href=/news/spark-0-9-0-released.htmlSpark 0.9.0 released/a span class=small(Feb 02, 2014)/span/li - lia href=/news/spark-0-8-1-released.htmlSpark 0.8.1 released/a - span class=small(Dec 19, 2013)/span/li - /ul p class=small style=text-align: right;a href=/news/index.htmlArchive/a/p /div Modified: spark/site/releases/spark-release-0-6-0.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-0-6-0.html?rev=1586127r1=1586126r2=1586127view=diff == --- spark/site/releases/spark-release-0-6-0.html (original) +++ spark/site/releases/spark-release-0-6-0.html Wed Apr 9 20:17:50 2014 @@ -124,6 +124,9 @@ h5Latest News/h5 ul class=list-unstyled + lia href=/news/spark-0-9-1-released.htmlSpark 0.9.1 released/a + span class=small(Apr 09, 2014)/span/li + lia href=/news/submit-talks-to-spark-summit-2014.htmlSubmissions and registration open for Spark Summit 2014/a span class=small(Mar 20, 2014)/span/li @@ -133,9 +136,6 @@ lia href=/news/spark-0-9-0-released.htmlSpark 0.9.0 released/a span class=small(Feb 02, 2014)/span/li - lia href=/news/spark-0-8-1-released.htmlSpark 0.8.1 released/a - span class=small(Dec 19, 2013)/span/li - /ul p class=small style=text-align: right;a href=/news/index.htmlArchive/a/p /div Modified: spark/site/releases/spark-release-0-6-1.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-0-6-1.html?rev=1586127r1=1586126r2=1586127view=diff == ---
svn commit: r1586167 [2/2] - in /spark/site/docs/0.9.1: ./ api/pyspark/
Modified: spark/site/docs/0.9.1/tuning.html
URL:
http://svn.apache.org/viewvc/spark/site/docs/0.9.1/tuning.html?rev=1586167r1=1586166r2=1586167view=diff
==
--- spark/site/docs/0.9.1/tuning.html (original)
+++ spark/site/docs/0.9.1/tuning.html Wed Apr 9 22:56:47 2014
@@ -25,6 +25,19 @@
link rel=stylesheet href=css/pygments-default.css
+!-- Google analytics script --
+script type=text/javascript
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-32518208-1']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+var ga = document.createElement('script'); ga.type =
'text/javascript'; ga.async = true;
+ga.src = ('https:' == document.location.protocol ? 'https://ssl' :
'http://www') + '.google-analytics.com/ga.js';
+var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s);
+ })();
+/script
+
/head
body
svn commit: r1586169 - in /spark: releases/_posts/2014-04-09-spark-release-0-9-1.md site/docs/0.9.1/0.9.1-docs.tgz site/releases/spark-release-0-9-1.html
Author: tdas Date: Wed Apr 9 23:19:13 2014 New Revision: 1586169 URL: http://svn.apache.org/r1586169 Log: Updated 0.9.1 release notes Removed: spark/site/docs/0.9.1/0.9.1-docs.tgz Modified: spark/releases/_posts/2014-04-09-spark-release-0-9-1.md spark/site/releases/spark-release-0-9-1.html Modified: spark/releases/_posts/2014-04-09-spark-release-0-9-1.md URL: http://svn.apache.org/viewvc/spark/releases/_posts/2014-04-09-spark-release-0-9-1.md?rev=1586169r1=1586168r2=1586169view=diff == --- spark/releases/_posts/2014-04-09-spark-release-0-9-1.md (original) +++ spark/releases/_posts/2014-04-09-spark-release-0-9-1.md Wed Apr 9 23:19:13 2014 @@ -59,7 +59,7 @@ Several bug fixes were made to YARN depl * Fixed bug in Python de-pickling [[SPARK-1135](https://issues.apache.org/jira/browse/SPARK-1135)] * Fixed bug in serialization of strings longer than 64K [[SPARK-1043](https://issues.apache.org/jira/browse/SPARK-1043)] * Fixed bug that made jobs hang when base file is not available [[SPARK-1025](https://issues.apache.org/jira/browse/SPARK-1025)] -* Added Missing RDD operations to PySpark - top, zip, foldByKey and setName +* Added Missing RDD operations to PySpark - top, zip, foldByKey, repartition, coallesce, getStorageLevel, setName and toDebugString ### Improvements to documentation * Streaming: Added documentation on running streaming application from spark-shell Modified: spark/site/releases/spark-release-0-9-1.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-0-9-1.html?rev=1586169r1=1586168r2=1586169view=diff == --- spark/site/releases/spark-release-0-9-1.html (original) +++ spark/site/releases/spark-release-0-9-1.html Wed Apr 9 23:19:13 2014 @@ -217,7 +217,7 @@ liFixed bug in Python de-pickling [a href=https://issues.apache.org/jira/browse/SPARK-1135;SPARK-1135/a]/li liFixed bug in serialization of strings longer than 64K [a href=https://issues.apache.org/jira/browse/SPARK-1043;SPARK-1043/a] /li liFixed bug that made jobs hang when base file is not available [a href=https://issues.apache.org/jira/browse/SPARK-1025;SPARK-1025/a] /li - liAdded Missing RDD operations to PySpark - top, zip, foldByKey and setName /li + liAdded Missing RDD operations to PySpark - top, zip, foldByKey, repartition, coallesce, getStorageLevel, setName and toDebugString/li /ul h3 id=improvements-to-documentationImprovements to documentation/h3
svn commit: r1586197 - in /spark: releases/_posts/2014-04-09-spark-release-0-9-1.md site/releases/spark-release-0-9-1.html
Author: tdas Date: Thu Apr 10 02:17:44 2014 New Revision: 1586197 URL: http://svn.apache.org/r1586197 Log: Fixed typo Modified: spark/releases/_posts/2014-04-09-spark-release-0-9-1.md spark/site/releases/spark-release-0-9-1.html Modified: spark/releases/_posts/2014-04-09-spark-release-0-9-1.md URL: http://svn.apache.org/viewvc/spark/releases/_posts/2014-04-09-spark-release-0-9-1.md?rev=1586197r1=1586196r2=1586197view=diff == --- spark/releases/_posts/2014-04-09-spark-release-0-9-1.md (original) +++ spark/releases/_posts/2014-04-09-spark-release-0-9-1.md Thu Apr 10 02:17:44 2014 @@ -59,7 +59,7 @@ Several bug fixes were made to YARN depl * Fixed bug in Python de-pickling [[SPARK-1135](https://issues.apache.org/jira/browse/SPARK-1135)] * Fixed bug in serialization of strings longer than 64K [[SPARK-1043](https://issues.apache.org/jira/browse/SPARK-1043)] * Fixed bug that made jobs hang when base file is not available [[SPARK-1025](https://issues.apache.org/jira/browse/SPARK-1025)] -* Added Missing RDD operations to PySpark - top, zip, foldByKey, repartition, coallesce, getStorageLevel, setName and toDebugString +* Added Missing RDD operations to PySpark - top, zip, foldByKey, repartition, coalesce, getStorageLevel, setName and toDebugString ### Improvements to documentation * Streaming: Added documentation on running streaming application from spark-shell Modified: spark/site/releases/spark-release-0-9-1.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-0-9-1.html?rev=1586197r1=1586196r2=1586197view=diff == --- spark/site/releases/spark-release-0-9-1.html (original) +++ spark/site/releases/spark-release-0-9-1.html Thu Apr 10 02:17:44 2014 @@ -217,7 +217,7 @@ liFixed bug in Python de-pickling [a href=https://issues.apache.org/jira/browse/SPARK-1135;SPARK-1135/a]/li liFixed bug in serialization of strings longer than 64K [a href=https://issues.apache.org/jira/browse/SPARK-1043;SPARK-1043/a] /li liFixed bug that made jobs hang when base file is not available [a href=https://issues.apache.org/jira/browse/SPARK-1025;SPARK-1025/a] /li - liAdded Missing RDD operations to PySpark - top, zip, foldByKey, repartition, coallesce, getStorageLevel, setName and toDebugString/li + liAdded Missing RDD operations to PySpark - top, zip, foldByKey, repartition, coalesce, getStorageLevel, setName and toDebugString/li /ul h3 id=improvements-to-documentationImprovements to documentation/h3
git commit: [SPARK-1592][streaming] Automatically remove streaming input blocks
Repository: spark
Updated Branches:
refs/heads/master 35e3d199f - 526a518bf
[SPARK-1592][streaming] Automatically remove streaming input blocks
The raw input data is stored as blocks in BlockManagers. Earlier they were
cleared by cleaner ttl. Now since streaming does not require cleaner TTL to be
set, the block would not get cleared. This increases up the Spark's memory
usage, which is not even accounted and shown in the Spark storage UI. It may
cause the data blocks to spill over to disk, which eventually slows down the
receiving of data (persisting to memory become bottlenecked by writing to disk).
The solution in this PR is to automatically remove those blocks. The mechanism
to keep track of which BlockRDDs (which has presents the raw data blocks as a
RDD) can be safely cleared already exists. Just use it to explicitly remove
blocks from BlockRDDs.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #512 from tdas/block-rdd-unpersist and squashes the following commits:
d25e610 [Tathagata Das] Merge remote-tracking branch 'apache/master' into
block-rdd-unpersist
5f46d69 [Tathagata Das] Merge remote-tracking branch 'apache/master' into
block-rdd-unpersist
2c320cd [Tathagata Das] Updated configuration with spark.streaming.unpersist
setting.
2d4b2fd [Tathagata Das] Automatically removed input blocks
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/526a518b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/526a518b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/526a518b
Branch: refs/heads/master
Commit: 526a518bf32ad55b926a26f16086f445fd0ae29f
Parents: 35e3d19
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu Apr 24 18:18:22 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Apr 24 18:18:22 2014 -0700
--
.../scala/org/apache/spark/rdd/BlockRDD.scala | 45 ++--
docs/configuration.md | 7 +-
.../scala/org/apache/spark/streaming/Time.scala | 2 +-
.../spark/streaming/dstream/DStream.scala | 16 -
.../spark/streaming/BasicOperationsSuite.scala | 76 +++-
.../spark/streaming/InputStreamsSuite.scala | 13
.../spark/streaming/NetworkReceiverSuite.scala | 1 +
7 files changed, 135 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/526a518b/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
--
diff --git a/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
b/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
index e6c4a6d..c64da88 100644
--- a/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
@@ -19,24 +19,30 @@ package org.apache.spark.rdd
import scala.reflect.ClassTag
-import org.apache.spark.{Partition, SparkContext, SparkEnv, TaskContext}
+import org.apache.spark._
import org.apache.spark.storage.{BlockId, BlockManager}
+import scala.Some
private[spark] class BlockRDDPartition(val blockId: BlockId, idx: Int) extends
Partition {
val index = idx
}
private[spark]
-class BlockRDD[T: ClassTag](sc: SparkContext, @transient blockIds:
Array[BlockId])
+class BlockRDD[T: ClassTag](@transient sc: SparkContext, @transient val
blockIds: Array[BlockId])
extends RDD[T](sc, Nil) {
@transient lazy val locations_ = BlockManager.blockIdsToHosts(blockIds,
SparkEnv.get)
+ @volatile private var _isValid = true
- override def getPartitions: Array[Partition] = (0 until blockIds.size).map(i
= {
-new BlockRDDPartition(blockIds(i), i).asInstanceOf[Partition]
- }).toArray
+ override def getPartitions: Array[Partition] = {
+assertValid()
+(0 until blockIds.size).map(i = {
+ new BlockRDDPartition(blockIds(i), i).asInstanceOf[Partition]
+}).toArray
+ }
override def compute(split: Partition, context: TaskContext): Iterator[T] = {
+assertValid()
val blockManager = SparkEnv.get.blockManager
val blockId = split.asInstanceOf[BlockRDDPartition].blockId
blockManager.get(blockId) match {
@@ -47,7 +53,36 @@ class BlockRDD[T: ClassTag](sc: SparkContext, @transient
blockIds: Array[BlockId
}
override def getPreferredLocations(split: Partition): Seq[String] = {
+assertValid()
locations_(split.asInstanceOf[BlockRDDPartition].blockId)
}
+
+ /**
+ * Remove the data blocks that this BlockRDD is made from. NOTE: This is an
+ * irreversible operation, as the data in the blocks cannot be recovered back
+ * once removed. Use it with caution.
+ */
+ private[spark] def removeBlocks() {
+blockIds.foreach { blockId =
+ sc.env.blockManager.master.removeBlock(blockId)
+}
+_isValid = false
git commit: [SPARK-1592][streaming] Automatically remove streaming input blocks
Repository: spark
Updated Branches:
refs/heads/branch-1.0 521d43599 - a3b6d8523
[SPARK-1592][streaming] Automatically remove streaming input blocks
The raw input data is stored as blocks in BlockManagers. Earlier they were
cleared by cleaner ttl. Now since streaming does not require cleaner TTL to be
set, the block would not get cleared. This increases up the Spark's memory
usage, which is not even accounted and shown in the Spark storage UI. It may
cause the data blocks to spill over to disk, which eventually slows down the
receiving of data (persisting to memory become bottlenecked by writing to disk).
The solution in this PR is to automatically remove those blocks. The mechanism
to keep track of which BlockRDDs (which has presents the raw data blocks as a
RDD) can be safely cleared already exists. Just use it to explicitly remove
blocks from BlockRDDs.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #512 from tdas/block-rdd-unpersist and squashes the following commits:
d25e610 [Tathagata Das] Merge remote-tracking branch 'apache/master' into
block-rdd-unpersist
5f46d69 [Tathagata Das] Merge remote-tracking branch 'apache/master' into
block-rdd-unpersist
2c320cd [Tathagata Das] Updated configuration with spark.streaming.unpersist
setting.
2d4b2fd [Tathagata Das] Automatically removed input blocks
(cherry picked from commit 526a518bf32ad55b926a26f16086f445fd0ae29f)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a3b6d852
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a3b6d852
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a3b6d852
Branch: refs/heads/branch-1.0
Commit: a3b6d852337c5c4c9afd4942699f35baaa6f691d
Parents: 521d435
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu Apr 24 18:18:22 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Apr 24 18:18:47 2014 -0700
--
.../scala/org/apache/spark/rdd/BlockRDD.scala | 45 ++--
docs/configuration.md | 7 +-
.../scala/org/apache/spark/streaming/Time.scala | 2 +-
.../spark/streaming/dstream/DStream.scala | 16 -
.../spark/streaming/BasicOperationsSuite.scala | 76 +++-
.../spark/streaming/InputStreamsSuite.scala | 13
.../spark/streaming/NetworkReceiverSuite.scala | 1 +
7 files changed, 135 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a3b6d852/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
--
diff --git a/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
b/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
index e6c4a6d..c64da88 100644
--- a/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
@@ -19,24 +19,30 @@ package org.apache.spark.rdd
import scala.reflect.ClassTag
-import org.apache.spark.{Partition, SparkContext, SparkEnv, TaskContext}
+import org.apache.spark._
import org.apache.spark.storage.{BlockId, BlockManager}
+import scala.Some
private[spark] class BlockRDDPartition(val blockId: BlockId, idx: Int) extends
Partition {
val index = idx
}
private[spark]
-class BlockRDD[T: ClassTag](sc: SparkContext, @transient blockIds:
Array[BlockId])
+class BlockRDD[T: ClassTag](@transient sc: SparkContext, @transient val
blockIds: Array[BlockId])
extends RDD[T](sc, Nil) {
@transient lazy val locations_ = BlockManager.blockIdsToHosts(blockIds,
SparkEnv.get)
+ @volatile private var _isValid = true
- override def getPartitions: Array[Partition] = (0 until blockIds.size).map(i
= {
-new BlockRDDPartition(blockIds(i), i).asInstanceOf[Partition]
- }).toArray
+ override def getPartitions: Array[Partition] = {
+assertValid()
+(0 until blockIds.size).map(i = {
+ new BlockRDDPartition(blockIds(i), i).asInstanceOf[Partition]
+}).toArray
+ }
override def compute(split: Partition, context: TaskContext): Iterator[T] = {
+assertValid()
val blockManager = SparkEnv.get.blockManager
val blockId = split.asInstanceOf[BlockRDDPartition].blockId
blockManager.get(blockId) match {
@@ -47,7 +53,36 @@ class BlockRDD[T: ClassTag](sc: SparkContext, @transient
blockIds: Array[BlockId
}
override def getPreferredLocations(split: Partition): Seq[String] = {
+assertValid()
locations_(split.asInstanceOf[BlockRDDPartition].blockId)
}
+
+ /**
+ * Remove the data blocks that this BlockRDD is made from. NOTE: This is an
+ * irreversible operation, as the data in the blocks cannot be recovered back
+ * once removed. Use it with caution.
+ */
+ private[spark] def
git commit: [SPARK-1510] Spark Streaming metrics source for metrics system
Repository: spark
Updated Branches:
refs/heads/master 44da5ab2d - 80429f3e2
[SPARK-1510] Spark Streaming metrics source for metrics system
This pulls in changes made by @jerryshao in
https://github.com/apache/spark/pull/424 and merges with the master.
Author: jerryshao saisai.s...@intel.com
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #545 from tdas/streaming-metrics and squashes the following commits:
034b443 [Tathagata Das] Merge remote-tracking branch 'apache-github/master'
into streaming-metrics
fb3b0a5 [jerryshao] Modify according master update
21939f5 [jerryshao] Style changes according to style check error
976116b [jerryshao] Add StreamSource in StreamingContext for better monitoring
through metrics system
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/80429f3e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/80429f3e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/80429f3e
Branch: refs/heads/master
Commit: 80429f3e2ab786d103297652922c3d8da3cf5a01
Parents: 44da5ab
Author: jerryshao saisai.s...@intel.com
Authored: Thu Apr 24 18:56:57 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Apr 24 18:56:57 2014 -0700
--
.../spark/streaming/StreamingContext.scala | 4 ++
.../spark/streaming/StreamingSource.scala | 73
.../ui/StreamingJobProgressListener.scala | 3 +-
3 files changed, 79 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/80429f3e/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
index 1c89543..e0677b7 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
@@ -154,6 +154,10 @@ class StreamingContext private[streaming] (
private[streaming] val uiTab = new StreamingTab(this)
+ /** Register streaming source to metrics system */
+ private val streamingSource = new StreamingSource(this)
+ SparkEnv.get.metricsSystem.registerSource(streamingSource)
+
/** Enumeration to identify current state of the StreamingContext */
private[streaming] object StreamingContextState extends Enumeration {
type CheckpointState = Value
http://git-wip-us.apache.org/repos/asf/spark/blob/80429f3e/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
b/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
new file mode 100644
index 000..774adc3
--- /dev/null
+++ b/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import com.codahale.metrics.{Gauge, MetricRegistry}
+
+import org.apache.spark.metrics.source.Source
+import org.apache.spark.streaming.ui.StreamingJobProgressListener
+
+private[streaming] class StreamingSource(ssc: StreamingContext) extends Source
{
+ val metricRegistry = new MetricRegistry
+ val sourceName = %s.StreamingMetrics.format(ssc.sparkContext.appName)
+
+ val streamingListener = ssc.uiTab.listener
+
+ private def registerGauge[T](name: String, f: StreamingJobProgressListener
= T,
+ defaultValue: T) {
+metricRegistry.register(MetricRegistry.name(streaming, name), new
Gauge[T] {
+ override def getValue: T =
Option(f(streamingListener)).getOrElse(defaultValue)
+})
+ }
+
+ // Gauge for number of network receivers
+ registerGauge(receivers, _.numReceivers, 0)
+
+ // Gauge for number of total completed batches
+ registerGauge(totalCompletedBatches
git commit: [SPARK-1510] Spark Streaming metrics source for metrics system
Repository: spark
Updated Branches:
refs/heads/branch-1.0 c8dd13221 - 0bc0f36d6
[SPARK-1510] Spark Streaming metrics source for metrics system
This pulls in changes made by @jerryshao in
https://github.com/apache/spark/pull/424 and merges with the master.
Author: jerryshao saisai.s...@intel.com
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #545 from tdas/streaming-metrics and squashes the following commits:
034b443 [Tathagata Das] Merge remote-tracking branch 'apache-github/master'
into streaming-metrics
fb3b0a5 [jerryshao] Modify according master update
21939f5 [jerryshao] Style changes according to style check error
976116b [jerryshao] Add StreamSource in StreamingContext for better monitoring
through metrics system
(cherry picked from commit 80429f3e2ab786d103297652922c3d8da3cf5a01)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0bc0f36d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0bc0f36d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0bc0f36d
Branch: refs/heads/branch-1.0
Commit: 0bc0f36d6be265d8ad5b1909fc06f03a661454a5
Parents: c8dd132
Author: jerryshao saisai.s...@intel.com
Authored: Thu Apr 24 18:56:57 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Apr 24 18:57:11 2014 -0700
--
.../spark/streaming/StreamingContext.scala | 4 ++
.../spark/streaming/StreamingSource.scala | 73
.../ui/StreamingJobProgressListener.scala | 3 +-
3 files changed, 79 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/0bc0f36d/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
index 1c89543..e0677b7 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
@@ -154,6 +154,10 @@ class StreamingContext private[streaming] (
private[streaming] val uiTab = new StreamingTab(this)
+ /** Register streaming source to metrics system */
+ private val streamingSource = new StreamingSource(this)
+ SparkEnv.get.metricsSystem.registerSource(streamingSource)
+
/** Enumeration to identify current state of the StreamingContext */
private[streaming] object StreamingContextState extends Enumeration {
type CheckpointState = Value
http://git-wip-us.apache.org/repos/asf/spark/blob/0bc0f36d/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
b/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
new file mode 100644
index 000..774adc3
--- /dev/null
+++ b/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import com.codahale.metrics.{Gauge, MetricRegistry}
+
+import org.apache.spark.metrics.source.Source
+import org.apache.spark.streaming.ui.StreamingJobProgressListener
+
+private[streaming] class StreamingSource(ssc: StreamingContext) extends Source
{
+ val metricRegistry = new MetricRegistry
+ val sourceName = %s.StreamingMetrics.format(ssc.sparkContext.appName)
+
+ val streamingListener = ssc.uiTab.listener
+
+ private def registerGauge[T](name: String, f: StreamingJobProgressListener
= T,
+ defaultValue: T) {
+metricRegistry.register(MetricRegistry.name(streaming, name), new
Gauge[T] {
+ override def getValue: T =
Option(f(streamingListener)).getOrElse(defaultValue)
+})
+ }
+
+ // Gauge for number of network receivers
+ registerGauge
git commit: [SPARK-1617] and [SPARK-1618] Improvements to streaming ui and bug fix to socket receiver
Repository: spark
Updated Branches:
refs/heads/master 968c0187a - cd12dd9bd
[SPARK-1617] and [SPARK-1618] Improvements to streaming ui and bug fix to
socket receiver
1617: These changes expose the receiver state (active or inactive) and last
error in the UI
1618: If the socket receiver cannot connect in the first attempt, it should try
to restart after a delay. That was broken, as the thread that restarts (hence,
stops) the receiver waited on Thread.join on itself!
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #540 from tdas/streaming-ui-fix and squashes the following commits:
e469434 [Tathagata Das] Merge remote-tracking branch 'apache-github/master'
into streaming-ui-fix
dbddf75 [Tathagata Das] Style fix.
66df1a5 [Tathagata Das] Merge remote-tracking branch 'apache/master' into
streaming-ui-fix
ad98bc9 [Tathagata Das] Refactored streaming listener to use ReceiverInfo.
d7f849c [Tathagata Das] Revert Moved BatchInfo from streaming.scheduler to
streaming.ui
5c80919 [Tathagata Das] Moved BatchInfo from streaming.scheduler to streaming.ui
da244f6 [Tathagata Das] Fixed socket receiver as well as made receiver state
and error visible in the streamign UI.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cd12dd9b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cd12dd9b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cd12dd9b
Branch: refs/heads/master
Commit: cd12dd9bde91303d0341180e5f70d2a03d6b65b6
Parents: 968c018
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu Apr 24 21:34:37 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Apr 24 21:34:37 2014 -0700
--
.../scala/org/apache/spark/ui/UIUtils.scala | 2 +-
.../streaming/dstream/SocketInputDStream.scala | 49 +++--
.../streaming/receiver/ActorReceiver.scala | 12 +++-
.../spark/streaming/receiver/Receiver.scala | 5 +-
.../streaming/receiver/ReceiverMessage.scala| 4 +-
.../streaming/receiver/ReceiverSupervisor.scala | 58
.../receiver/ReceiverSupervisorImpl.scala | 24 +---
.../spark/streaming/scheduler/BatchInfo.scala | 3 +
.../streaming/scheduler/ReceiverInfo.scala | 37 +
.../streaming/scheduler/ReceiverTracker.scala | 40 +-
.../streaming/scheduler/StreamingListener.scala | 25 -
.../ui/StreamingJobProgressListener.scala | 18 +-
.../spark/streaming/ui/StreamingPage.scala | 20 +--
.../spark/streaming/NetworkReceiverSuite.scala | 8 ++-
.../streaming/StreamingListenerSuite.scala | 15 ++---
15 files changed, 217 insertions(+), 103 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/cd12dd9b/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
--
diff --git a/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
b/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
index cf987a1..a3d6a18 100644
--- a/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
+++ b/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
@@ -122,7 +122,7 @@ private[spark] object UIUtils extends Logging {
}
}
if (unit.isEmpty) {
- %d.formatLocal(Locale.US, value)
+ %d.formatLocal(Locale.US, value.toInt)
} else {
%.1f%s.formatLocal(Locale.US, value, unit)
}
http://git-wip-us.apache.org/repos/asf/spark/blob/cd12dd9b/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
index 1e32727..8b72bcf 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
@@ -50,49 +50,42 @@ class SocketReceiver[T: ClassTag](
storageLevel: StorageLevel
) extends Receiver[T](storageLevel) with Logging {
- var socket: Socket = null
- var receivingThread: Thread = null
-
def onStart() {
-receivingThread = new Thread(Socket Receiver) {
- override def run() {
-connect()
-receive()
- }
-}
-receivingThread.start()
+// Start the thread that receives data over a connection
+new Thread(Socket Receiver) {
+ setDaemon(true)
+ override def run() { receive() }
+}.start()
}
def onStop() {
-if (socket != null) {
- socket.close()
-}
-socket = null
-if (receivingThread != null) {
- receivingThread.join()
-}
+// There is nothing much to do
git commit: [SPARK-1617] and [SPARK-1618] Improvements to streaming ui and bug fix to socket receiver
Repository: spark
Updated Branches:
refs/heads/branch-1.0 777a9a5a1 - d933c710f
[SPARK-1617] and [SPARK-1618] Improvements to streaming ui and bug fix to
socket receiver
1617: These changes expose the receiver state (active or inactive) and last
error in the UI
1618: If the socket receiver cannot connect in the first attempt, it should try
to restart after a delay. That was broken, as the thread that restarts (hence,
stops) the receiver waited on Thread.join on itself!
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #540 from tdas/streaming-ui-fix and squashes the following commits:
e469434 [Tathagata Das] Merge remote-tracking branch 'apache-github/master'
into streaming-ui-fix
dbddf75 [Tathagata Das] Style fix.
66df1a5 [Tathagata Das] Merge remote-tracking branch 'apache/master' into
streaming-ui-fix
ad98bc9 [Tathagata Das] Refactored streaming listener to use ReceiverInfo.
d7f849c [Tathagata Das] Revert Moved BatchInfo from streaming.scheduler to
streaming.ui
5c80919 [Tathagata Das] Moved BatchInfo from streaming.scheduler to streaming.ui
da244f6 [Tathagata Das] Fixed socket receiver as well as made receiver state
and error visible in the streamign UI.
(cherry picked from commit cd12dd9bde91303d0341180e5f70d2a03d6b65b6)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d933c710
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d933c710
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d933c710
Branch: refs/heads/branch-1.0
Commit: d933c710f9b49f398446d51859fe6953d2e4b6f9
Parents: 777a9a5
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu Apr 24 21:34:37 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Apr 24 21:34:50 2014 -0700
--
.../scala/org/apache/spark/ui/UIUtils.scala | 2 +-
.../streaming/dstream/SocketInputDStream.scala | 49 +++--
.../streaming/receiver/ActorReceiver.scala | 12 +++-
.../spark/streaming/receiver/Receiver.scala | 5 +-
.../streaming/receiver/ReceiverMessage.scala| 4 +-
.../streaming/receiver/ReceiverSupervisor.scala | 58
.../receiver/ReceiverSupervisorImpl.scala | 24 +---
.../spark/streaming/scheduler/BatchInfo.scala | 3 +
.../streaming/scheduler/ReceiverInfo.scala | 37 +
.../streaming/scheduler/ReceiverTracker.scala | 40 +-
.../streaming/scheduler/StreamingListener.scala | 25 -
.../ui/StreamingJobProgressListener.scala | 18 +-
.../spark/streaming/ui/StreamingPage.scala | 20 +--
.../spark/streaming/NetworkReceiverSuite.scala | 8 ++-
.../streaming/StreamingListenerSuite.scala | 15 ++---
15 files changed, 217 insertions(+), 103 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/d933c710/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
--
diff --git a/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
b/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
index cf987a1..a3d6a18 100644
--- a/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
+++ b/core/src/main/scala/org/apache/spark/ui/UIUtils.scala
@@ -122,7 +122,7 @@ private[spark] object UIUtils extends Logging {
}
}
if (unit.isEmpty) {
- %d.formatLocal(Locale.US, value)
+ %d.formatLocal(Locale.US, value.toInt)
} else {
%.1f%s.formatLocal(Locale.US, value, unit)
}
http://git-wip-us.apache.org/repos/asf/spark/blob/d933c710/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
index 1e32727..8b72bcf 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala
@@ -50,49 +50,42 @@ class SocketReceiver[T: ClassTag](
storageLevel: StorageLevel
) extends Receiver[T](storageLevel) with Logging {
- var socket: Socket = null
- var receivingThread: Thread = null
-
def onStart() {
-receivingThread = new Thread(Socket Receiver) {
- override def run() {
-connect()
-receive()
- }
-}
-receivingThread.start()
+// Start the thread that receives data over a connection
+new Thread(Socket Receiver) {
+ setDaemon(true)
+ override def run() { receive() }
+}.start()
}
def onStop() {
-if (socket != null) {
- socket.close
git commit: [Spark-1382] Fix NPE in DStream.slice (updated version of #365)
Repository: spark
Updated Branches:
refs/heads/branch-1.0 94c71e0a2 - f85c6815e
[Spark-1382] Fix NPE in DStream.slice (updated version of #365)
@zsxwing I cherry-picked your changes and merged the master. #365 had some
conflicts once again!
Author: zsxwing zsxw...@gmail.com
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #562 from tdas/SPARK-1382 and squashes the following commits:
e2962c1 [Tathagata Das] Merge remote-tracking branch 'apache-github/master'
into SPARK-1382
20968d9 [zsxwing] Replace Exception with SparkException in DStream
e476651 [zsxwing] Merge remote-tracking branch 'origin/master' into SPARK-1382
35ba56a [zsxwing] SPARK-1382: Fix NPE in DStream.slice
(cherry picked from commit 058797c1722c9251f6bc6ad2672cb0e79146b04f)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f85c6815
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f85c6815
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f85c6815
Branch: refs/heads/branch-1.0
Commit: f85c6815ebb80cdc35e63c7e3c2a354a808e4249
Parents: 94c71e0
Author: zsxwing zsxw...@gmail.com
Authored: Fri Apr 25 19:04:34 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Fri Apr 25 19:04:45 2014 -0700
--
.../spark/streaming/dstream/DStream.scala | 22 +++-
.../spark/streaming/BasicOperationsSuite.scala | 12 ++-
2 files changed, 23 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/f85c6815/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
index f69f69e..4709a62 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
@@ -18,20 +18,19 @@
package org.apache.spark.streaming.dstream
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
import scala.deprecated
import scala.collection.mutable.HashMap
import scala.reflect.ClassTag
-import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
-
-import org.apache.spark.Logging
+import org.apache.spark.{Logging, SparkException}
import org.apache.spark.rdd.{BlockRDD, RDD}
import org.apache.spark.storage.StorageLevel
-import org.apache.spark.util.MetadataCleaner
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.scheduler.Job
-import org.apache.spark.streaming.Duration
+import org.apache.spark.util.MetadataCleaner
/**
* A Discretized Stream (DStream), the basic abstraction in Spark Streaming,
is a continuous
@@ -144,7 +143,7 @@ abstract class DStream[T: ClassTag] (
*/
private[streaming] def initialize(time: Time) {
if (zeroTime != null zeroTime != time) {
- throw new Exception(ZeroTime is already initialized to + zeroTime
+ throw new SparkException(ZeroTime is already initialized to + zeroTime
+ , cannot initialize it again to + time)
}
zeroTime = time
@@ -220,7 +219,7 @@ abstract class DStream[T: ClassTag] (
which requires + this.getClass.getSimpleName + to remember
generated RDDs for more +
than + rememberDuration.milliseconds / 1000 + seconds. But
Spark's metadata cleanup +
delay is set to + metadataCleanerDelay + seconds, which is not
sufficient. Please +
-set the Java property 'spark.cleaner.delay' to more than +
+set the Java cleaner delay to more than +
math.ceil(rememberDuration.milliseconds / 1000.0).toInt + seconds.
)
@@ -235,7 +234,7 @@ abstract class DStream[T: ClassTag] (
private[streaming] def setContext(s: StreamingContext) {
if (ssc != null ssc != s) {
- throw new Exception(Context is already set in + this + , cannot set
it again)
+ throw new SparkException(Context is already set in + this + , cannot
set it again)
}
ssc = s
logInfo(Set context for + this)
@@ -244,7 +243,7 @@ abstract class DStream[T: ClassTag] (
private[streaming] def setGraph(g: DStreamGraph) {
if (graph != null graph != g) {
- throw new Exception(Graph is already set in + this + , cannot set it
again)
+ throw new SparkException(Graph is already set in + this + , cannot
set it again)
}
graph = g
dependencies.foreach(_.setGraph(graph))
@@ -261,7 +260,7 @@ abstract class DStream[T: ClassTag] (
/** Checks whether the 'time' is valid wrt slideDuration for generating RDD
git commit: [SPARK-1504], [SPARK-1505], [SPARK-1558] Updated Spark Streaming guide
Repository: spark Updated Branches: refs/heads/master 3292e2a71 - a975a19f2 [SPARK-1504], [SPARK-1505], [SPARK-1558] Updated Spark Streaming guide - SPARK-1558: Updated custom receiver guide to match it with the new API - SPARK-1504: Added deployment and monitoring subsection to streaming - SPARK-1505: Added migration guide for migrating from 0.9.x and below to Spark 1.0 - Updated various Java streaming examples to use JavaReceiverInputDStream to highlight the API change. - Removed the requirement for cleaner ttl from streaming guide Author: Tathagata Das tathagata.das1...@gmail.com Closes #652 from tdas/doc-fix and squashes the following commits: cb4f4b7 [Tathagata Das] Possible fix for flaky graceful shutdown test. ab71f7f [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into doc-fix 8d6ff9b [Tathagata Das] Addded migration guide to Spark Streaming. 7d171df [Tathagata Das] Added reference to JavaReceiverInputStream in examples and streaming guide. 49edd7c [Tathagata Das] Change java doc links to use Java docs. 11528d7 [Tathagata Das] Updated links on index page. ff80970 [Tathagata Das] More updates to streaming guide. 4dc42e9 [Tathagata Das] Added monitoring and other documentation in the streaming guide. 14c6564 [Tathagata Das] Updated custom receiver guide. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a975a19f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a975a19f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a975a19f Branch: refs/heads/master Commit: a975a19f21e71f448b3fdb2ed4461e28ef439900 Parents: 3292e2a Author: Tathagata Das tathagata.das1...@gmail.com Authored: Mon May 5 15:28:19 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 5 15:28:19 2014 -0700 -- docs/configuration.md | 6 +- docs/index.md | 6 +- docs/streaming-custom-receivers.md | 273 +-- docs/streaming-programming-guide.md | 200 +++--- .../streaming/examples/JavaCustomReceiver.java | 3 +- .../streaming/examples/JavaFlumeEventCount.java | 2 +- .../streaming/examples/JavaKafkaWordCount.java | 4 +- .../examples/JavaNetworkWordCount.java | 5 +- .../spark/streaming/receiver/Receiver.scala | 6 +- .../spark/streaming/StreamingContextSuite.scala | 1 + 10 files changed, 360 insertions(+), 146 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/a975a19f/docs/configuration.md -- diff --git a/docs/configuration.md b/docs/configuration.md index 81ad895..d6f316b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -462,7 +462,7 @@ Apart from these, the following properties are also available, and may be useful td(infinite)/td td Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks generated, etc.). -Periodic cleanups will ensure that metadata older than this duration will be forgetten. This is +Periodic cleanups will ensure that metadata older than this duration will be forgotten. This is useful for running Spark for many hours / days (for example, running 24/7 in case of Spark Streaming applications). Note that any RDD that persists in memory for more than this duration will be cleared as well. /td @@ -471,8 +471,8 @@ Apart from these, the following properties are also available, and may be useful tdspark.streaming.blockInterval/td td200/td td -Duration (milliseconds) of how long to batch new objects coming from network receivers used -in Spark Streaming. +Interval (milliseconds) at which data received by Spark Streaming receivers is coalesced +into blocks of data before storing them in Spark. /td /tr tr http://git-wip-us.apache.org/repos/asf/spark/blob/a975a19f/docs/index.md -- diff --git a/docs/index.md b/docs/index.md index 6fc9a4f..2daa208 100644 --- a/docs/index.md +++ b/docs/index.md @@ -112,10 +112,10 @@ Note that on Windows, you need to set the environment variables on separate line * [Shark](http://shark.cs.berkeley.edu): Apache Hive over Spark * [Mailing Lists](http://spark.apache.org/mailing-lists.html): ask questions about Spark here * [AMP Camps](http://ampcamp.berkeley.edu/): a series of training camps at UC Berkeley that featured talks and - exercises about Spark, Shark, Mesos, and more. [Videos](http://ampcamp.berkeley.edu/agenda-2012), - [slides](http://ampcamp.berkeley.edu/agenda-2012) and [exercises](http://ampcamp.berkeley.edu/exercises-2012) are + exercises about Spark, Shark, Spark Streaming, Mesos, and more
git commit: [SPARK-1504], [SPARK-1505], [SPARK-1558] Updated Spark Streaming guide
Repository: spark Updated Branches: refs/heads/branch-1.0 80f4360e7 - 1fac4ecbd [SPARK-1504], [SPARK-1505], [SPARK-1558] Updated Spark Streaming guide - SPARK-1558: Updated custom receiver guide to match it with the new API - SPARK-1504: Added deployment and monitoring subsection to streaming - SPARK-1505: Added migration guide for migrating from 0.9.x and below to Spark 1.0 - Updated various Java streaming examples to use JavaReceiverInputDStream to highlight the API change. - Removed the requirement for cleaner ttl from streaming guide Author: Tathagata Das tathagata.das1...@gmail.com Closes #652 from tdas/doc-fix and squashes the following commits: cb4f4b7 [Tathagata Das] Possible fix for flaky graceful shutdown test. ab71f7f [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into doc-fix 8d6ff9b [Tathagata Das] Addded migration guide to Spark Streaming. 7d171df [Tathagata Das] Added reference to JavaReceiverInputStream in examples and streaming guide. 49edd7c [Tathagata Das] Change java doc links to use Java docs. 11528d7 [Tathagata Das] Updated links on index page. ff80970 [Tathagata Das] More updates to streaming guide. 4dc42e9 [Tathagata Das] Added monitoring and other documentation in the streaming guide. 14c6564 [Tathagata Das] Updated custom receiver guide. (cherry picked from commit a975a19f21e71f448b3fdb2ed4461e28ef439900) Signed-off-by: Tathagata Das tathagata.das1...@gmail.com Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1fac4ecb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1fac4ecb Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1fac4ecb Branch: refs/heads/branch-1.0 Commit: 1fac4ecbdd0c15992fd75372cbd7fec24244d21b Parents: 80f4360 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Mon May 5 15:28:19 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 5 15:28:54 2014 -0700 -- docs/configuration.md | 6 +- docs/index.md | 6 +- docs/streaming-custom-receivers.md | 273 +-- docs/streaming-programming-guide.md | 200 +++--- .../streaming/examples/JavaCustomReceiver.java | 3 +- .../streaming/examples/JavaFlumeEventCount.java | 2 +- .../streaming/examples/JavaKafkaWordCount.java | 4 +- .../examples/JavaNetworkWordCount.java | 5 +- .../spark/streaming/receiver/Receiver.scala | 6 +- .../spark/streaming/StreamingContextSuite.scala | 1 + 10 files changed, 360 insertions(+), 146 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/1fac4ecb/docs/configuration.md -- diff --git a/docs/configuration.md b/docs/configuration.md index 81ad895..d6f316b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -462,7 +462,7 @@ Apart from these, the following properties are also available, and may be useful td(infinite)/td td Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks generated, etc.). -Periodic cleanups will ensure that metadata older than this duration will be forgetten. This is +Periodic cleanups will ensure that metadata older than this duration will be forgotten. This is useful for running Spark for many hours / days (for example, running 24/7 in case of Spark Streaming applications). Note that any RDD that persists in memory for more than this duration will be cleared as well. /td @@ -471,8 +471,8 @@ Apart from these, the following properties are also available, and may be useful tdspark.streaming.blockInterval/td td200/td td -Duration (milliseconds) of how long to batch new objects coming from network receivers used -in Spark Streaming. +Interval (milliseconds) at which data received by Spark Streaming receivers is coalesced +into blocks of data before storing them in Spark. /td /tr tr http://git-wip-us.apache.org/repos/asf/spark/blob/1fac4ecb/docs/index.md -- diff --git a/docs/index.md b/docs/index.md index 6fc9a4f..2daa208 100644 --- a/docs/index.md +++ b/docs/index.md @@ -112,10 +112,10 @@ Note that on Windows, you need to set the environment variables on separate line * [Shark](http://shark.cs.berkeley.edu): Apache Hive over Spark * [Mailing Lists](http://spark.apache.org/mailing-lists.html): ask questions about Spark here * [AMP Camps](http://ampcamp.berkeley.edu/): a series of training camps at UC Berkeley that featured talks and - exercises about Spark, Shark, Mesos, and more. [Videos](http://ampcamp.berkeley.edu/agenda-2012), - [slides](http://ampcamp.berkeley.edu/agenda
git commit: Fixed streaming examples docs to use run-example instead of spark-submit
Repository: spark
Updated Branches:
refs/heads/branch-1.0 69ec3149f - c7571d8c6
Fixed streaming examples docs to use run-example instead of spark-submit
Pretty self-explanatory
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #722 from tdas/example-fix and squashes the following commits:
7839979 [Tathagata Das] Minor changes.
0673441 [Tathagata Das] Fixed java docs of java streaming example
e687123 [Tathagata Das] Fixed scala style errors.
9b8d112 [Tathagata Das] Fixed streaming examples docs to use run-example
instead of spark-submit.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c7571d8c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c7571d8c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c7571d8c
Branch: refs/heads/branch-1.0
Commit: c7571d8c6ba058b67cca2b910fd0efacc06642cd
Parents: 69ec314
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Wed May 14 04:17:32 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Wed May 14 04:24:48 2014 -0700
--
.../examples/streaming/JavaCustomReceiver.java | 13 ++---
.../examples/streaming/JavaFlumeEventCount.java | 6 +-
.../examples/streaming/JavaKafkaWordCount.java | 6 +-
.../streaming/JavaNetworkWordCount.java | 13 +++--
.../examples/streaming/ActorWordCount.scala | 6 +-
.../examples/streaming/CustomReceiver.scala | 19 +++
.../examples/streaming/FlumeEventCount.scala| 9 ++-
.../examples/streaming/HdfsWordCount.scala | 5 +-
.../examples/streaming/KafkaWordCount.scala | 6 +-
.../examples/streaming/MQTTWordCount.scala | 10 ++--
.../examples/streaming/NetworkWordCount.scala | 14 +++--
.../streaming/RecoverableNetworkWordCount.scala | 7 +--
.../streaming/StatefulNetworkWordCount.scala| 6 +-
.../examples/streaming/TwitterPopularTags.scala | 22 +++-
.../examples/streaming/ZeroMQWordCount.scala| 8 +--
.../clickstream/PageViewGenerator.scala | 10 ++--
.../streaming/clickstream/PageViewStream.scala | 7 ++-
.../streaming/twitter/TwitterInputDStream.scala | 58
18 files changed, 130 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c7571d8c/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java
--
diff --git
a/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java
b/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java
index 7f558f3..5622df5 100644
---
a/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java
+++
b/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java
@@ -19,6 +19,7 @@ package org.apache.spark.examples.streaming;
import com.google.common.collect.Lists;
+import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
@@ -48,25 +49,23 @@ import java.util.regex.Pattern;
* To run this on your local machine, you need to first run a Netcat server
*`$ nc -lk `
* and then run the example
- *`$ ./run org.apache.spark.examples.streaming.JavaCustomReceiver local[2]
localhost `
+ *`$ bin/run-example
org.apache.spark.examples.streaming.JavaCustomReceiver localhost `
*/
public class JavaCustomReceiver extends ReceiverString {
private static final Pattern SPACE = Pattern.compile( );
public static void main(String[] args) {
-if (args.length 3) {
- System.err.println(Usage: JavaNetworkWordCount master hostname
port\n +
- In local mode, master should be 'local[n]' with n 1);
+if (args.length 2) {
+ System.err.println(Usage: JavaNetworkWordCount hostname port);
System.exit(1);
}
StreamingExamples.setStreamingLogLevels();
// Create the context with a 1 second batch size
-JavaStreamingContext ssc = new JavaStreamingContext(args[0],
JavaNetworkWordCount,
-new Duration(1000), System.getenv(SPARK_HOME),
-JavaStreamingContext.jarOfClass(JavaNetworkWordCount.class));
+SparkConf sparkConf = new SparkConf().setAppName(JavaCustomReceiver);
+JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, new
Duration(1000));
// Create a input stream with the custom receiver on target ip:port and
count the
// words in input stream of \n delimited text (eg. generated by 'nc')
http://git-wip-us.apache.org/repos/asf/spark/blob/c7571d8c/examples/src/main/java/org/apache/spark/examples/streaming/JavaFlumeEventCount.java
git commit: Converted bang to ask to avoid scary warning when a block is removed
Repository: spark Updated Branches: refs/heads/branch-1.0 1d56cd544 - b8c17e392 Converted bang to ask to avoid scary warning when a block is removed Removing a block through the blockmanager gave a scary warning messages in the driver. ``` 2014-05-08 20:16:19,172 WARN BlockManagerMasterActor: Got unknown message: true 2014-05-08 20:16:19,172 WARN BlockManagerMasterActor: Got unknown message: true 2014-05-08 20:16:19,172 WARN BlockManagerMasterActor: Got unknown message: true ``` This is because the [BlockManagerSlaveActor](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManagerSlaveActor.scala#L44) would send back an acknowledgement (true). But the BlockManagerMasterActor would have sent the RemoveBlock message as a send, not as ask(), so would reject the receiver true as a unknown message. @pwendell Author: Tathagata Das tathagata.das1...@gmail.com Closes #708 from tdas/bm-fix and squashes the following commits: ed4ef15 [Tathagata Das] Converted bang to ask to avoid scary warning when a block is removed. (cherry picked from commit 32868f31f88aebd580ab9329dc51a30c26af7a74) Signed-off-by: Tathagata Das tathagata.das1...@gmail.com Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b8c17e39 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b8c17e39 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b8c17e39 Branch: refs/heads/branch-1.0 Commit: b8c17e3928d070d4757d44995516b8872196e5c9 Parents: 1d56cd5 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Thu May 8 22:34:08 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Thu May 8 22:34:21 2014 -0700 -- .../scala/org/apache/spark/storage/BlockManagerMasterActor.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/b8c17e39/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala -- diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala index 98fa0df..6aed322 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala @@ -250,7 +250,7 @@ class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf, listenerBus // Remove the block from the slave's BlockManager. // Doesn't actually wait for a confirmation and the message might get lost. // If message loss becomes frequent, we should add retry logic here. - blockManager.get.slaveActor ! RemoveBlock(blockId) + blockManager.get.slaveActor.ask(RemoveBlock(blockId))(akkaTimeout) } } }
git commit: Converted bang to ask to avoid scary warning when a block is removed
Repository: spark Updated Branches: refs/heads/master 4c60fd1e8 - 32868f31f Converted bang to ask to avoid scary warning when a block is removed Removing a block through the blockmanager gave a scary warning messages in the driver. ``` 2014-05-08 20:16:19,172 WARN BlockManagerMasterActor: Got unknown message: true 2014-05-08 20:16:19,172 WARN BlockManagerMasterActor: Got unknown message: true 2014-05-08 20:16:19,172 WARN BlockManagerMasterActor: Got unknown message: true ``` This is because the [BlockManagerSlaveActor](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManagerSlaveActor.scala#L44) would send back an acknowledgement (true). But the BlockManagerMasterActor would have sent the RemoveBlock message as a send, not as ask(), so would reject the receiver true as a unknown message. @pwendell Author: Tathagata Das tathagata.das1...@gmail.com Closes #708 from tdas/bm-fix and squashes the following commits: ed4ef15 [Tathagata Das] Converted bang to ask to avoid scary warning when a block is removed. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/32868f31 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/32868f31 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/32868f31 Branch: refs/heads/master Commit: 32868f31f88aebd580ab9329dc51a30c26af7a74 Parents: 4c60fd1 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Thu May 8 22:34:08 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Thu May 8 22:34:08 2014 -0700 -- .../scala/org/apache/spark/storage/BlockManagerMasterActor.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/32868f31/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala -- diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala index 98fa0df..6aed322 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala @@ -250,7 +250,7 @@ class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf, listenerBus // Remove the block from the slave's BlockManager. // Doesn't actually wait for a confirmation and the message might get lost. // If message loss becomes frequent, we should add retry logic here. - blockManager.get.slaveActor ! RemoveBlock(blockId) + blockManager.get.slaveActor.ask(RemoveBlock(blockId))(akkaTimeout) } } }
git commit: [SPARK-1876] Windows fixes to deal with latest distribution layout changes
Repository: spark Updated Branches: refs/heads/master df0aa8353 - 7b70a7071 [SPARK-1876] Windows fixes to deal with latest distribution layout changes - Look for JARs in the right place - Launch examples the same way as on Unix - Load datanucleus JARs if they exist - Don't attempt to parse local paths as URIs in SparkSubmit, since paths with C:\ are not valid URIs - Also fixed POM exclusion rules for datanucleus (it wasn't properly excluding it, whereas SBT was) Author: Matei Zaharia ma...@databricks.com Closes #819 from mateiz/win-fixes and squashes the following commits: d558f96 [Matei Zaharia] Fix comment 228577b [Matei Zaharia] Review comments d3b71c7 [Matei Zaharia] Properly exclude datanucleus files in Maven assembly 144af84 [Matei Zaharia] Update Windows scripts to match latest binary package layout Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7b70a707 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7b70a707 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7b70a707 Branch: refs/heads/master Commit: 7b70a7071894dd90ea1d0091542b3e13e7ef8d3a Parents: df0aa83 Author: Matei Zaharia ma...@databricks.com Authored: Mon May 19 15:02:35 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 19 15:02:35 2014 -0700 -- README.md | 7 +-- assembly/pom.xml| 2 +- bin/compute-classpath.cmd | 24 - bin/run-example | 23 + bin/run-example2.cmd| 51 +++- bin/spark-class2.cmd| 2 + .../org/apache/spark/deploy/SparkSubmit.scala | 2 +- 7 files changed, 81 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/7b70a707/README.md -- diff --git a/README.md b/README.md index 9c2e32b..6211a58 100644 --- a/README.md +++ b/README.md @@ -9,13 +9,14 @@ You can find the latest Spark documentation, including a programming guide, on the project webpage at http://spark.apache.org/documentation.html. This README file only contains basic setup instructions. - ## Building Spark Spark is built on Scala 2.10. To build Spark and its example programs, run: ./sbt/sbt assembly +(You do not need to do this if you downloaded a pre-built package.) + ## Interactive Scala Shell The easiest way to start using Spark is through the Scala shell: @@ -41,9 +42,9 @@ And run the following command, which should also return 1000: Spark also comes with several sample programs in the `examples` directory. To run one of them, use `./bin/run-example class [params]`. For example: -./bin/run-example org.apache.spark.examples.SparkLR +./bin/run-example SparkPi -will run the Logistic Regression example locally. +will run the Pi example locally. You can set the MASTER environment variable when running examples to submit examples to a cluster. This can be a mesos:// or spark:// URL, http://git-wip-us.apache.org/repos/asf/spark/blob/7b70a707/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index abd8935..963357b 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -96,7 +96,7 @@ filter artifact*:*/artifact excludes -excludeorg.datanucleus:*/exclude +excludeorg/datanucleus/**/exclude excludeMETA-INF/*.SF/exclude excludeMETA-INF/*.DSA/exclude excludeMETA-INF/*.RSA/exclude http://git-wip-us.apache.org/repos/asf/spark/blob/7b70a707/bin/compute-classpath.cmd -- diff --git a/bin/compute-classpath.cmd b/bin/compute-classpath.cmd index 065553e..58710cd 100644 --- a/bin/compute-classpath.cmd +++ b/bin/compute-classpath.cmd @@ -20,6 +20,13 @@ rem rem This script computes Spark's classpath and prints it to stdout; it's used by both the run rem script and the ExecutorRunner in standalone cluster mode. +rem If we're called from spark-class2.cmd, it already set enabledelayedexpansion and setting +rem it here would stop us from affecting its copy of the CLASSPATH variable; otherwise we +rem need to set it here because we use !datanucleus_jars! below. +if %DONT_PRINT_CLASSPATH%==1 goto skip_delayed_expansion +setlocal enabledelayedexpansion +:skip_delayed_expansion + set SCALA_VERSION=2.10 rem Figure out where the Spark framework is installed @@ -31,7 +38,7 @@ if exist %FWDIR%conf\spark-env.cmd call %FWDIR%conf\spark-env.cmd rem Build up classpath set
git commit: SPARK-1878: Fix the incorrect initialization order
Repository: spark
Updated Branches:
refs/heads/branch-1.0 111c121ae - 901102c1b
SPARK-1878: Fix the incorrect initialization order
JIRA: https://issues.apache.org/jira/browse/SPARK-1878
Author: zsxwing zsxw...@gmail.com
Closes #822 from zsxwing/SPARK-1878 and squashes the following commits:
4a47e27 [zsxwing] SPARK-1878: Fix the incorrect initialization order
(cherry picked from commit 1811ba8ccb580979aa2e12019e6a82805f09ab53)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/901102c1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/901102c1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/901102c1
Branch: refs/heads/branch-1.0
Commit: 901102c1ba5f800705819916f2b7a38b6750cffb
Parents: 111c121
Author: zsxwing zsxw...@gmail.com
Authored: Mon May 19 16:41:31 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 16:41:53 2014 -0700
--
.../apache/spark/streaming/api/java/JavaStreamingContext.scala | 6 +++---
.../src/test/java/org/apache/spark/streaming/JavaAPISuite.java | 4
2 files changed, 7 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/901102c1/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
b/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
index 75a3e93..18605ca 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
@@ -142,12 +142,12 @@ class JavaStreamingContext(val ssc: StreamingContext) {
*/
def this(path: String, hadoopConf: Configuration) = this(new
StreamingContext(path, hadoopConf))
- @deprecated(use sparkContext, 0.9.0)
- val sc: JavaSparkContext = sparkContext
-
/** The underlying SparkContext */
val sparkContext = new JavaSparkContext(ssc.sc)
+ @deprecated(use sparkContext, 0.9.0)
+ val sc: JavaSparkContext = sparkContext
+
/**
* Create an input stream from network source hostname:port. Data is
received using
* a TCP socket and the receive bytes is interpreted as UTF8 encoded \n
delimited
http://git-wip-us.apache.org/repos/asf/spark/blob/901102c1/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
--
diff --git
a/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
b/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
index ce58cb1..4efeb8d 100644
--- a/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
+++ b/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
@@ -55,6 +55,10 @@ public class JavaAPISuite extends LocalJavaStreamingContext
implements Serializa
equalIterator(a.iterator(), b.iterator());
}
+ @Test
+ public void testInitialization() {
+Assert.assertNotNull(ssc.sc());
+ }
@SuppressWarnings(unchecked)
@Test
git commit: SPARK-1879. Increase MaxPermSize since some of our builds have many classes
Repository: spark
Updated Branches:
refs/heads/master 1811ba8cc - 5af99d761
SPARK-1879. Increase MaxPermSize since some of our builds have many classes
See https://issues.apache.org/jira/browse/SPARK-1879 -- builds with Hadoop2 and
Hive ran out of PermGen space in spark-shell, when those things added up with
the Scala compiler.
Note that users can still override it by setting their own Java options with
this change. Their options will come later in the command string than the
-XX:MaxPermSize=128m.
Author: Matei Zaharia ma...@databricks.com
Closes #823 from mateiz/spark-1879 and squashes the following commits:
6bc0ee8 [Matei Zaharia] Increase MaxPermSize to 128m since some of our builds
have lots of classes
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5af99d76
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5af99d76
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5af99d76
Branch: refs/heads/master
Commit: 5af99d7617ba3b9fbfdb345ef9571b7dd41f45a1
Parents: 1811ba8
Author: Matei Zaharia ma...@databricks.com
Authored: Mon May 19 18:42:28 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 18:42:28 2014 -0700
--
bin/spark-class | 4 ++--
bin/spark-class2.cmd | 4 ++--
.../main/scala/org/apache/spark/deploy/worker/CommandUtils.scala | 4 +++-
3 files changed, 7 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/5af99d76/bin/spark-class
--
diff --git a/bin/spark-class b/bin/spark-class
index 6480ccb..2e57295 100755
--- a/bin/spark-class
+++ b/bin/spark-class
@@ -99,14 +99,14 @@ else
fi
# Set JAVA_OPTS to be able to load native libraries and to set heap size
-JAVA_OPTS=$OUR_JAVA_OPTS
+JAVA_OPTS=-XX:MaxPermSize=128m $OUR_JAVA_OPTS
JAVA_OPTS=$JAVA_OPTS -Xms$OUR_JAVA_MEM -Xmx$OUR_JAVA_MEM
# Load extra JAVA_OPTS from conf/java-opts, if it exists
if [ -e $FWDIR/conf/java-opts ] ; then
JAVA_OPTS=$JAVA_OPTS `cat $FWDIR/conf/java-opts`
fi
export JAVA_OPTS
-# Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in ExecutorRunner.scala!
+# Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in CommandUtils.scala!
if [ ! -f $FWDIR/RELEASE ]; then
# Exit if the user hasn't compiled Spark
http://git-wip-us.apache.org/repos/asf/spark/blob/5af99d76/bin/spark-class2.cmd
--
diff --git a/bin/spark-class2.cmd b/bin/spark-class2.cmd
index 266edd9..e420eb4 100755
--- a/bin/spark-class2.cmd
+++ b/bin/spark-class2.cmd
@@ -77,8 +77,8 @@ rem All drivers use SPARK_JAVA_OPTS + SPARK_DRIVER_MEMORY.
The repl also uses SP
)
rem Set JAVA_OPTS to be able to load native libraries and to set heap size
-set JAVA_OPTS=%OUR_JAVA_OPTS% -Djava.library.path=%SPARK_LIBRARY_PATH%
-Xms%OUR_JAVA_MEM% -Xmx%OUR_JAVA_MEM%
-rem Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in ExecutorRunner.scala!
+set JAVA_OPTS=-XX:MaxPermSize=128m %OUR_JAVA_OPTS%
-Djava.library.path=%SPARK_LIBRARY_PATH% -Xms%OUR_JAVA_MEM% -Xmx%OUR_JAVA_MEM%
+rem Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in CommandUtils.scala!
rem Test whether the user has built Spark
if exist %FWDIR%RELEASE goto skip_build_test
http://git-wip-us.apache.org/repos/asf/spark/blob/5af99d76/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
index c7f0f24..4af5bc3 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
@@ -65,6 +65,8 @@ object CommandUtils extends Logging {
Seq()
}
+val permGenOpt = Seq(-XX:MaxPermSize=128m)
+
// Figure out our classpath with the external compute-classpath script
val ext = if (System.getProperty(os.name).startsWith(Windows)) .cmd
else .sh
val classPath = Utils.executeAndGetOutput(
@@ -73,7 +75,7 @@ object CommandUtils extends Logging {
val userClassPath = command.classPathEntries ++ Seq(classPath)
Seq(-cp,
userClassPath.filterNot(_.isEmpty).mkString(File.pathSeparator)) ++
- libraryOpts ++ extraOpts ++ workerLocalOpts ++ memoryOpts
+ permGenOpt ++ libraryOpts ++ extraOpts ++ workerLocalOpts ++ memoryOpts
}
/** Spawn a thread that will
git commit: SPARK-1689: Spark application should die when removed by Master
Repository: spark
Updated Branches:
refs/heads/master 6a2c5c610 - b0ce22e07
SPARK-1689: Spark application should die when removed by Master
scheduler.error() will mask the error if there are active tasks. Being removed
is a cataclysmic event for Spark applications, and should probably be treated
as such.
Author: Aaron Davidson aa...@databricks.com
Closes #832 from aarondav/i-love-u and squashes the following commits:
9f1200f [Aaron Davidson] SPARK-1689: Spark application should die when removed
by Master
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b0ce22e0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b0ce22e0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b0ce22e0
Branch: refs/heads/master
Commit: b0ce22e071da4cc62ec5e29abf7b1299b8e4a6b0
Parents: 6a2c5c6
Author: Aaron Davidson aa...@databricks.com
Authored: Mon May 19 20:55:26 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 20:55:26 2014 -0700
--
.../spark/scheduler/cluster/SparkDeploySchedulerBackend.scala | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b0ce22e0/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
index 9768670..9c07b3f 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
@@ -88,6 +88,8 @@ private[spark] class SparkDeploySchedulerBackend(
if (!stopping) {
logError(Application has been killed. Reason: + reason)
scheduler.error(reason)
+ // Ensure the application terminates, as we can no longer run jobs.
+ sc.stop()
}
}
git commit: SPARK-1689: Spark application should die when removed by Master
Repository: spark
Updated Branches:
refs/heads/branch-1.0 875c54fb3 - 78b6e6f1e
SPARK-1689: Spark application should die when removed by Master
scheduler.error() will mask the error if there are active tasks. Being removed
is a cataclysmic event for Spark applications, and should probably be treated
as such.
Author: Aaron Davidson aa...@databricks.com
Closes #832 from aarondav/i-love-u and squashes the following commits:
9f1200f [Aaron Davidson] SPARK-1689: Spark application should die when removed
by Master
(cherry picked from commit b0ce22e071da4cc62ec5e29abf7b1299b8e4a6b0)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/78b6e6f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/78b6e6f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/78b6e6f1
Branch: refs/heads/branch-1.0
Commit: 78b6e6f1e8ee6a27ef4eed93aac6eba716b5ffce
Parents: 875c54f
Author: Aaron Davidson aa...@databricks.com
Authored: Mon May 19 20:55:26 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 21:01:18 2014 -0700
--
.../spark/scheduler/cluster/SparkDeploySchedulerBackend.scala | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/78b6e6f1/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
index 9768670..9c07b3f 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
@@ -88,6 +88,8 @@ private[spark] class SparkDeploySchedulerBackend(
if (!stopping) {
logError(Application has been killed. Reason: + reason)
scheduler.error(reason)
+ // Ensure the application terminates, as we can no longer run jobs.
+ sc.stop()
}
}
[4/4] git commit: [SPARK-1874][MLLIB] Clean up MLlib sample data
[SPARK-1874][MLLIB] Clean up MLlib sample data 1. Added synthetic datasets for `MovieLensALS`, `LinearRegression`, `BinaryClassification`. 2. Embedded instructions in the help message of those example apps. Per discussion with Matei on the JIRA page, new example data is under `data/mllib`. Author: Xiangrui Meng m...@databricks.com Closes #833 from mengxr/mllib-sample-data and squashes the following commits: 59f0a18 [Xiangrui Meng] add sample binary classification data 3c2f92f [Xiangrui Meng] add linear regression data 050f1ca [Xiangrui Meng] add a sample dataset for MovieLensALS example Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bcb9dce6 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bcb9dce6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bcb9dce6 Branch: refs/heads/master Commit: bcb9dce6f444a977c714117811bce0c54b417650 Parents: b0ce22e Author: Xiangrui Meng m...@databricks.com Authored: Mon May 19 21:29:33 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 19 21:29:33 2014 -0700 -- .../mllib/sample_binary_classification_data.txt | 100 ++ data/mllib/sample_linear_regression_data.txt| 501 ++ data/mllib/sample_movielens_data.txt| 1501 ++ .../examples/mllib/BinaryClassification.scala | 12 +- .../spark/examples/mllib/LinearRegression.scala | 11 +- .../spark/examples/mllib/MovieLensALS.scala | 15 + 6 files changed, 2138 insertions(+), 2 deletions(-) --
[1/4] [SPARK-1874][MLLIB] Clean up MLlib sample data
Repository: spark Updated Branches: refs/heads/branch-1.0 78b6e6f1e - 1c6c8b5bd http://git-wip-us.apache.org/repos/asf/spark/blob/1c6c8b5b/data/mllib/sample_movielens_data.txt -- diff --git a/data/mllib/sample_movielens_data.txt b/data/mllib/sample_movielens_data.txt new file mode 100644 index 000..f0eee19 --- /dev/null +++ b/data/mllib/sample_movielens_data.txt @@ -0,0 +1,1501 @@ +0::2::3 +0::3::1 +0::5::2 +0::9::4 +0::11::1 +0::12::2 +0::15::1 +0::17::1 +0::19::1 +0::21::1 +0::23::1 +0::26::3 +0::27::1 +0::28::1 +0::29::1 +0::30::1 +0::31::1 +0::34::1 +0::37::1 +0::41::2 +0::44::1 +0::45::2 +0::46::1 +0::47::1 +0::48::1 +0::50::1 +0::51::1 +0::54::1 +0::55::1 +0::59::2 +0::61::2 +0::64::1 +0::67::1 +0::68::1 +0::69::1 +0::71::1 +0::72::1 +0::77::2 +0::79::1 +0::83::1 +0::87::1 +0::89::2 +0::91::3 +0::92::4 +0::94::1 +0::95::2 +0::96::1 +0::98::1 +0::99::1 +1::2::2 +1::3::1 +1::4::2 +1::6::1 +1::9::3 +1::12::1 +1::13::1 +1::14::1 +1::16::1 +1::19::1 +1::21::3 +1::27::1 +1::28::3 +1::33::1 +1::36::2 +1::37::1 +1::40::1 +1::41::2 +1::43::1 +1::44::1 +1::47::1 +1::50::1 +1::54::1 +1::56::2 +1::57::1 +1::58::1 +1::60::1 +1::62::4 +1::63::1 +1::67::1 +1::68::4 +1::70::2 +1::72::1 +1::73::1 +1::74::2 +1::76::1 +1::77::3 +1::78::1 +1::81::1 +1::82::1 +1::85::3 +1::86::2 +1::88::2 +1::91::1 +1::92::2 +1::93::1 +1::94::2 +1::96::1 +1::97::1 +2::4::3 +2::6::1 +2::8::5 +2::9::1 +2::10::1 +2::12::3 +2::13::1 +2::15::2 +2::18::2 +2::19::4 +2::22::1 +2::26::1 +2::28::1 +2::34::4 +2::35::1 +2::37::5 +2::38::1 +2::39::5 +2::40::4 +2::47::1 +2::50::1 +2::52::2 +2::54::1 +2::55::1 +2::57::2 +2::58::2 +2::59::1 +2::61::1 +2::62::1 +2::64::1 +2::65::1 +2::66::3 +2::68::1 +2::71::3 +2::76::1 +2::77::1 +2::78::1 +2::80::1 +2::83::5 +2::85::1 +2::87::2 +2::88::1 +2::89::4 +2::90::1 +2::92::4 +2::93::5 +3::0::1 +3::1::1 +3::2::1 +3::7::3 +3::8::3 +3::9::1 +3::14::1 +3::15::1 +3::16::1 +3::18::4 +3::19::1 +3::24::3 +3::26::1 +3::29::3 +3::33::1 +3::34::3 +3::35::1 +3::36::3 +3::37::1 +3::38::2 +3::43::1 +3::44::1 +3::46::1 +3::47::1 +3::51::5 +3::52::3 +3::56::1 +3::58::1 +3::60::3 +3::62::1 +3::65::2 +3::66::1 +3::67::1 +3::68::2 +3::70::1 +3::72::2 +3::76::3 +3::79::3 +3::80::4 +3::81::1 +3::83::1 +3::84::1 +3::86::1 +3::87::2 +3::88::4 +3::89::1 +3::91::1 +3::94::3 +4::1::1 +4::6::1 +4::8::1 +4::9::1 +4::10::1 +4::11::1 +4::12::1 +4::13::1 +4::14::2 +4::15::1 +4::17::1 +4::20::1 +4::22::1 +4::23::1 +4::24::1 +4::29::4 +4::30::1 +4::31::1 +4::34::1 +4::35::1 +4::36::1 +4::39::2 +4::40::3 +4::41::4 +4::43::2 +4::44::1 +4::45::1 +4::46::1 +4::47::1 +4::49::2 +4::50::1 +4::51::1 +4::52::4 +4::54::1 +4::55::1 +4::60::3 +4::61::1 +4::62::4 +4::63::3 +4::65::1 +4::67::2 +4::69::1 +4::70::4 +4::71::1 +4::73::1 +4::78::1 +4::84::1 +4::85::1 +4::87::3 +4::88::3 +4::89::2 +4::96::1 +4::97::1 +4::98::1 +4::99::1 +5::0::1 +5::1::1 +5::4::1 +5::5::1 +5::8::1 +5::9::3 +5::10::2 +5::13::3 +5::15::1 +5::19::1 +5::20::3 +5::21::2 +5::23::3 +5::27::1 +5::28::1 +5::29::1 +5::31::1 +5::36::3 +5::38::2 +5::39::1 +5::42::1 +5::48::3 +5::49::4 +5::50::3 +5::51::1 +5::52::1 +5::54::1 +5::55::5 +5::56::3 +5::58::1 +5::60::1 +5::61::1 +5::64::3 +5::65::2 +5::68::4 +5::70::1 +5::71::1 +5::72::1 +5::74::1 +5::79::1 +5::81::2 +5::84::1 +5::85::1 +5::86::1 +5::88::1 +5::90::4 +5::91::2 +5::95::2 +5::99::1 +6::0::1 +6::1::1 +6::2::3 +6::5::1 +6::6::1 +6::9::1 +6::10::1 +6::15::2 +6::16::2 +6::17::1 +6::18::1 +6::20::1 +6::21::1 +6::22::1 +6::24::1 +6::25::5 +6::26::1 +6::28::1 +6::30::1 +6::33::1 +6::38::1 +6::39::1 +6::43::4 +6::44::1 +6::45::1 +6::48::1 +6::49::1 +6::50::1 +6::53::1 +6::54::1 +6::55::1 +6::56::1 +6::58::4 +6::59::1 +6::60::1 +6::61::3 +6::63::3 +6::66::1 +6::67::3 +6::68::1 +6::69::1 +6::71::2 +6::73::1 +6::75::1 +6::77::1 +6::79::1 +6::81::1 +6::84::1 +6::85::3 +6::86::1 +6::87::1 +6::88::1 +6::89::1 +6::91::2 +6::94::1 +6::95::2 +6::96::1 +7::1::1 +7::2::2 +7::3::1 +7::4::1 +7::7::1 +7::10::1 +7::11::2 +7::14::2 +7::15::1 +7::16::1 +7::18::1 +7::21::1 +7::22::1 +7::23::1 +7::25::5 +7::26::1 +7::29::4 +7::30::1 +7::31::3 +7::32::1 +7::33::1 +7::35::1 +7::37::2 +7::39::3 +7::40::2 +7::42::2 +7::44::1 +7::45::2 +7::47::4 +7::48::1 +7::49::1 +7::53::1 +7::54::1 +7::55::1 +7::56::1 +7::59::1 +7::61::2 +7::62::3 +7::63::2 +7::66::1 +7::67::3 +7::74::1 +7::75::1 +7::76::3 +7::77::1 +7::81::1 +7::82::1 +7::84::2 +7::85::4 +7::86::1 +7::92::2 +7::96::1 +7::97::1 +7::98::1 +8::0::1 +8::2::4 +8::3::2 +8::4::2 +8::5::1 +8::7::1 +8::9::1 +8::11::1 +8::15::1 +8::18::1 +8::19::1 +8::21::1 +8::29::5 +8::31::3 +8::33::1 +8::35::1 +8::36::1 +8::40::2 +8::44::1 +8::45::1 +8::50::1 +8::51::1 +8::52::5 +8::53::5 +8::54::1 +8::55::1 +8::56::1 +8::58::4 +8::60::3 +8::62::4 +8::64::1 +8::67::3 +8::69::1 +8::71::1 +8::72::3 +8::77::3 +8::78::1 +8::79::1 +8::83::1 +8::85::5 +8::86::1 +8::88::1 +8::90::1 +8::92::2 +8::95::4 +8::96::3 +8::97::1 +8::98::1 +8::99::1 +9::2::3 +9::3::1
git commit: [Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Repository: spark
Updated Branches:
refs/heads/branch-1.0 1c6c8b5bd - 6cbe2a37c
[Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Updated version of #821
Author: Tathagata Das tathagata.das1...@gmail.com
Author: Ghidireac bogd...@u448a5b0a73d45358d94a.ant.amazon.com
Closes #835 from tdas/SPARK-1877 and squashes the following commits:
f346f71 [Tathagata Das] Addressed Patrick's comments.
fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with
serialized objects
(cherry picked from commit 52eb54d02403a3c37d84b9da7cc1cdb261048cf8)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6cbe2a37
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6cbe2a37
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6cbe2a37
Branch: refs/heads/branch-1.0
Commit: 6cbe2a37ccb14f65b6d6b813a585adbbc43684c4
Parents: 1c6c8b5
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Mon May 19 22:36:24 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 22:36:37 2014 -0700
--
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6cbe2a37/core/src/main/scala/org/apache/spark/SparkContext.scala
--
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 634c10c..49737fa 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -718,7 +718,7 @@ class SparkContext(config: SparkConf) extends Logging {
minPartitions: Int = defaultMinPartitions
): RDD[T] = {
sequenceFile(path, classOf[NullWritable], classOf[BytesWritable],
minPartitions)
- .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes))
+ .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes,
Utils.getContextOrSparkClassLoader))
}
protected[spark] def checkpointFile[T: ClassTag](
git commit: [Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Repository: spark
Updated Branches:
refs/heads/master bcb9dce6f - 52eb54d02
[Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Updated version of #821
Author: Tathagata Das tathagata.das1...@gmail.com
Author: Ghidireac bogd...@u448a5b0a73d45358d94a.ant.amazon.com
Closes #835 from tdas/SPARK-1877 and squashes the following commits:
f346f71 [Tathagata Das] Addressed Patrick's comments.
fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with
serialized objects
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/52eb54d0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/52eb54d0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/52eb54d0
Branch: refs/heads/master
Commit: 52eb54d02403a3c37d84b9da7cc1cdb261048cf8
Parents: bcb9dce
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Mon May 19 22:36:24 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 22:36:24 2014 -0700
--
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/52eb54d0/core/src/main/scala/org/apache/spark/SparkContext.scala
--
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 634c10c..49737fa 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -718,7 +718,7 @@ class SparkContext(config: SparkConf) extends Logging {
minPartitions: Int = defaultMinPartitions
): RDD[T] = {
sequenceFile(path, classOf[NullWritable], classOf[BytesWritable],
minPartitions)
- .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes))
+ .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes,
Utils.getContextOrSparkClassLoader))
}
protected[spark] def checkpointFile[T: ClassTag](
[2/2] git commit: Revert [maven-release-plugin] prepare release v1.0.0-rc9
Revert [maven-release-plugin] prepare release v1.0.0-rc9 This reverts commit 920f947eb5a22a679c0c3186cf69ee75f6041c75. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0d988421 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0d988421 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0d988421 Branch: refs/heads/branch-1.0 Commit: 0d988421742bf43fbd13531fa7ede8d93e59a19b Parents: 3f3e988 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Mon May 19 23:15:20 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 19 23:15:20 2014 -0700 -- assembly/pom.xml | 6 +++--- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 24 insertions(+), 24 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/0d988421/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index bed4261..963357b 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0/version +version1.0.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent @@ -122,8 +122,8 @@ transformer implementation=org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer resourcelog4j.properties/resource /transformer -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer / -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer / +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer/ +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer/ /transformers /configuration /execution http://git-wip-us.apache.org/repos/asf/spark/blob/0d988421/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 08932bb..355f437 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0/version +version1.0.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/0d988421/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 3e22641..bab50f5 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0/version +version1.0.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/0d988421/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 006757a..874bcd7 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0/version +version1.0.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/0d988421/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 3ba984e..6aec215 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0/version +version1.0.0-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent
[1/2] git commit: Revert [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.0 1c00f2a25 - 0d9884217 Revert [maven-release-plugin] prepare for next development iteration This reverts commit f8e611955096c5c1c7db5764b9d2851b1d295f0d. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3f3e988c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3f3e988c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3f3e988c Branch: refs/heads/branch-1.0 Commit: 3f3e988cab4ac350f79ae3e2aadbfd0b5e6938e9 Parents: 1c00f2a Author: Tathagata Das tathagata.das1...@gmail.com Authored: Mon May 19 23:13:45 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 19 23:13:45 2014 -0700 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/3f3e988c/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 720ca77..bed4261 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/3f3e988c/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 85f6d99..08932bb 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/3f3e988c/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 47c2507..3e22641 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/3f3e988c/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index b7cbb1a..006757a 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/3f3e988c/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index b8fc07f..3ba984e 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/3f3e988c/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 9eeb2e1..cb4dd47 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/3f3e988c/external/mqtt/pom.xml -- diff --git
git commit: [Hotfix] Blacklisted flaky HiveCompatibility test
Repository: spark
Updated Branches:
refs/heads/branch-1.0 0d9884217 - b4d93d38d
[Hotfix] Blacklisted flaky HiveCompatibility test
`lateral_view_outer` query sometimes returns a different set of 10 rows.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #838 from tdas/hive-test-fix2 and squashes the following commits:
9128a0d [Tathagata Das] Blacklisted flaky HiveCompatibility test.
(cherry picked from commit 7f0cfe47f4709843d70ceccc25dee7551206ce0d)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b4d93d38
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b4d93d38
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b4d93d38
Branch: refs/heads/branch-1.0
Commit: b4d93d38d9da61721e64919f95447fafe87bf4d1
Parents: 0d98842
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Tue May 20 10:27:12 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Tue May 20 10:27:28 2014 -0700
--
.../spark/sql/hive/execution/HiveCompatibilitySuite.scala | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b4d93d38/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
--
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
index d83732b..9031abf 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
@@ -169,7 +169,10 @@ class HiveCompatibilitySuite extends HiveQueryFileTest
with BeforeAndAfter {
archive_corrupt,
// No support for case sensitivity is resolution using hive properties atm.
-case_sensitivity
+case_sensitivity,
+
+// Flaky test, Hive sometimes returns different set of 10 rows.
+lateral_view_outer
)
/**
@@ -476,7 +479,6 @@ class HiveCompatibilitySuite extends HiveQueryFileTest with
BeforeAndAfter {
join_view,
lateral_view,
lateral_view_cp,
-lateral_view_outer,
lateral_view_ppd,
lineage1,
literal_double,
git commit: [SPARK-1870] Make spark-submit --jars work in yarn-cluster mode.
Repository: spark
Updated Branches:
refs/heads/branch-1.0 75af8bd33 - 79cd26c52
[SPARK-1870] Make spark-submit --jars work in yarn-cluster mode.
Sent secondary jars to distributed cache of all containers and add the cached
jars to classpath before executors start. Tested on a YARN cluster (CDH-5.0).
`spark-submit --jars` also works in standalone server and `yarn-client`. Thanks
for @andrewor14 for testing!
I removed Doesn't work for drivers in standalone mode with cluster deploy
mode. from `spark-submit`'s help message, though we haven't tested mesos yet.
CC: @dbtsai @sryza
Author: Xiangrui Meng m...@databricks.com
Closes #848 from mengxr/yarn-classpath and squashes the following commits:
23e7df4 [Xiangrui Meng] rename spark.jar to __spark__.jar and app.jar to
__app__.jar to avoid confliction apped $CWD/ and $CWD/* to the classpath remove
unused methods
a40f6ed [Xiangrui Meng] standalone - cluster
65e04ad [Xiangrui Meng] update spark-submit help message and add a comment for
yarn-client
11e5354 [Xiangrui Meng] minor changes
3e7e1c4 [Xiangrui Meng] use sparkConf instead of hadoop conf
dc3c825 [Xiangrui Meng] add secondary jars to classpath in yarn
(cherry picked from commit dba314029b4c9d72d7e48a2093b39edd01931f57)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/79cd26c5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/79cd26c5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/79cd26c5
Branch: refs/heads/branch-1.0
Commit: 79cd26c528902369b9a2354ba0a0d033e0cf5950
Parents: 75af8bd
Author: Xiangrui Meng m...@databricks.com
Authored: Thu May 22 01:52:50 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu May 22 01:53:45 2014 -0700
--
.../spark/deploy/SparkSubmitArguments.scala | 3 +-
.../apache/spark/deploy/yarn/ClientBase.scala | 69 +---
.../cluster/YarnClientSchedulerBackend.scala| 2 +-
3 files changed, 19 insertions(+), 55 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/79cd26c5/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
index 264d454..0cc05fb 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
@@ -326,8 +326,7 @@ private[spark] class SparkSubmitArguments(args:
Seq[String]) {
| --class CLASS_NAME Your application's main class (for Java
/ Scala apps).
| --name NAME A name of your application.
| --jars JARS Comma-separated list of local jars to
include on the driver
-| and executor classpaths. Doesn't work
for drivers in
-| standalone mode with cluster deploy
mode.
+| and executor classpaths.
| --py-files PY_FILES Comma-separated list of .zip or .egg
files to place on the
| PYTHONPATH for Python apps.
| --files FILES Comma-separated list of files to be
placed in the working
http://git-wip-us.apache.org/repos/asf/spark/blob/79cd26c5/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
--
diff --git
a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
index 27a518c..aeb3f00 100644
--- a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
+++ b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
@@ -44,7 +44,7 @@ import org.apache.spark.{Logging, SparkConf, SparkContext}
* Client submits an application to the YARN ResourceManager.
*
* Depending on the deployment mode this will launch one of two application
master classes:
- * 1. In standalone mode, it will launch an
[[org.apache.spark.deploy.yarn.ApplicationMaster]]
+ * 1. In cluster mode, it will launch an
[[org.apache.spark.deploy.yarn.ApplicationMaster]]
* which launches a driver program inside of the cluster.
* 2. In client mode, it will launch an
[[org.apache.spark.deploy.yarn.ExecutorLauncher]] to
* request executors on behalf of a driver running outside of the cluster.
@@ -220,10 +220,11 @@ trait ClientBase extends Logging {
}
}
+var cachedSecondaryJarLinks = ListBuffer.empty[String]
git commit: [SPARK-1897] Respect spark.jars (and --jars) in spark-shell
Repository: spark
Updated Branches:
refs/heads/branch-1.0 a56621622 - 23cc40e39
[SPARK-1897] Respect spark.jars (and --jars) in spark-shell
Spark shell currently overwrites `spark.jars` with `ADD_JARS`. In all modes
except yarn-cluster, this means the `--jar` flag passed to `bin/spark-shell` is
also discarded. However, in the
[docs](http://people.apache.org/~pwendell/spark-1.0.0-rc7-docs/scala-programming-guide.html#initializing-spark),
we explicitly tell the users to add the jars this way.
Author: Andrew Or andrewo...@gmail.com
Closes #849 from andrewor14/shell-jars and squashes the following commits:
928a7e6 [Andrew Or] ',' - , (minor)
afc357c [Andrew Or] Handle spark.jars == in SparkILoop, not SparkSubmit
c6da113 [Andrew Or] Do not set spark.jars to
d8549f7 [Andrew Or] Respect spark.jars and --jars in spark-shell
(cherry picked from commit 8edbee7d1b4afc192d97ba192a5526affc464205)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/23cc40e3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/23cc40e3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/23cc40e3
Branch: refs/heads/branch-1.0
Commit: 23cc40e39acc598816ac46b36463e13587a0dd60
Parents: a566216
Author: Andrew Or andrewo...@gmail.com
Authored: Thu May 22 20:25:41 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu May 22 20:25:53 2014 -0700
--
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/23cc40e3/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
--
diff --git a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
index 296da74..55684e9 100644
--- a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
+++ b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
@@ -993,7 +993,13 @@ object SparkILoop {
implicit def loopToInterpreter(repl: SparkILoop): SparkIMain = repl.intp
private def echo(msg: String) = Console println msg
- def getAddedJars: Array[String] =
Option(System.getenv(ADD_JARS)).map(_.split(',')).getOrElse(new
Array[String](0))
+ def getAddedJars: Array[String] = {
+val envJars = sys.env.get(ADD_JARS)
+val propJars = sys.props.get(spark.jars).flatMap { p =
+ if (p == ) None else Some(p)
+}
+propJars.orElse(envJars).map(_.split(,)).getOrElse(Array.empty)
+ }
// Designed primarily for use by test code: take a String with a
// bunch of code, and prints out a transcript of what it would look
git commit: [SPARK-1896] Respect spark.master (and --master) before MASTER in spark-shell
Repository: spark
Updated Branches:
refs/heads/master 8edbee7d1 - cce77457e
[SPARK-1896] Respect spark.master (and --master) before MASTER in spark-shell
The hierarchy for configuring the Spark master in the shell is as follows:
```
MASTER --master spark.master (spark-defaults.conf)
```
This is inconsistent with the way we run normal applications, which is:
```
--master spark.master (spark-defaults.conf) MASTER
```
I was trying to run a shell locally on a standalone cluster launched through
the ec2 scripts, which automatically set `MASTER` in spark-env.sh. It was
surprising to me that `--master` didn't take effect, considering that this is
the way we tell users to set their masters
[here](http://people.apache.org/~pwendell/spark-1.0.0-rc7-docs/scala-programming-guide.html#initializing-spark).
Author: Andrew Or andrewo...@gmail.com
Closes #846 from andrewor14/shell-master and squashes the following commits:
2cb81c9 [Andrew Or] Respect spark.master before MASTER in REPL
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cce77457
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cce77457
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cce77457
Branch: refs/heads/master
Commit: cce77457e00aa5f1f4db3d50454cf257efb156ed
Parents: 8edbee7
Author: Andrew Or andrewo...@gmail.com
Authored: Thu May 22 20:32:27 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu May 22 20:32:27 2014 -0700
--
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/cce77457/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
--
diff --git a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
index 55684e9..5f34362 100644
--- a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
+++ b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
@@ -962,11 +962,10 @@ class SparkILoop(in0: Option[BufferedReader], protected
val out: JPrintWriter,
private def getMaster(): String = {
val master = this.master match {
case Some(m) = m
- case None = {
+ case None =
val envMaster = sys.env.get(MASTER)
val propMaster = sys.props.get(spark.master)
-envMaster.orElse(propMaster).getOrElse(local[*])
- }
+propMaster.orElse(envMaster).getOrElse(local[*])
}
master
}
git commit: Updated scripts for auditing releases
Repository: spark
Updated Branches:
refs/heads/master cce77457e - b2bdd0e50
Updated scripts for auditing releases
- Added script to automatically generate change list CHANGES.txt
- Added test for verifying linking against maven distributions of `spark-sql`
and `spark-hive`
- Added SBT projects for testing functionality of `spark-sql` and `spark-hive`
- Fixed issues in existing tests that might have come up because of changes in
Spark 1.0
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #844 from tdas/update-dev-scripts and squashes the following commits:
25090ba [Tathagata Das] Added missing license
e2e20b3 [Tathagata Das] Updated tests for auditing releases.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b2bdd0e5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b2bdd0e5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b2bdd0e5
Branch: refs/heads/master
Commit: b2bdd0e505f1ae3d39c46139f17bd43779ece635
Parents: cce7745
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu May 22 20:48:55 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu May 22 20:48:55 2014 -0700
--
dev/audit-release/audit_release.py | 11 +-
.../maven_app_core/src/main/java/SimpleApp.java | 1 +
.../sbt_app_core/src/main/scala/SparkApp.scala | 2 +-
dev/audit-release/sbt_app_hive/build.sbt| 29 +++
dev/audit-release/sbt_app_hive/data.txt | 9 +
.../src/main/resources/hive-site.xml| 213 +++
.../sbt_app_hive/src/main/scala/HiveApp.scala | 57 +
dev/audit-release/sbt_app_sql/build.sbt | 29 +++
.../sbt_app_sql/src/main/scala/SqlApp.scala | 57 +
.../src/main/scala/StreamingApp.scala | 1 -
dev/create-release/generate-changelist.py | 144 +
11 files changed, 547 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b2bdd0e5/dev/audit-release/audit_release.py
--
diff --git a/dev/audit-release/audit_release.py
b/dev/audit-release/audit_release.py
index 4a816d4..8c7573b 100755
--- a/dev/audit-release/audit_release.py
+++ b/dev/audit-release/audit_release.py
@@ -93,9 +93,12 @@ original_dir = os.getcwd()
# For each of these modules, we'll test an 'empty' application in sbt and
# maven that links against them. This will catch issues with messed up
# dependencies within those projects.
-modules = [spark-core, spark-bagel, spark-mllib, spark-streaming,
spark-repl,
- spark-graphx, spark-streaming-flume, spark-streaming-kafka,
- spark-streaming-mqtt, spark-streaming-twitter,
spark-streaming-zeromq]
+modules = [
+ spark-core, spark-bagel, spark-mllib, spark-streaming, spark-repl,
+ spark-graphx, spark-streaming-flume, spark-streaming-kafka,
+ spark-streaming-mqtt, spark-streaming-twitter, spark-streaming-zeromq,
+ spark-catalyst, spark-sql, spark-hive
+]
modules = map(lambda m: %s_%s % (m, SCALA_BINARY_VERSION), modules)
# Check for directories that might interfere with tests
@@ -122,7 +125,7 @@ for module in modules:
os.chdir(original_dir)
# SBT application tests
-for app in [sbt_app_core, sbt_app_graphx, sbt_app_streaming]:
+for app in [sbt_app_core, sbt_app_graphx, sbt_app_streaming,
sbt_app_sql, sbt_app_hive]:
os.chdir(app)
ret = run_cmd(sbt clean run, exit_on_failure=False)
test(ret == 0, sbt application (%s) % app)
http://git-wip-us.apache.org/repos/asf/spark/blob/b2bdd0e5/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
--
diff --git a/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
b/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
index 6b65dda..5217689 100644
--- a/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
+++ b/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
@@ -37,5 +37,6 @@ public class SimpleApp {
System.exit(-1);
}
System.out.println(Test succeeded);
+ sc.stop();
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/b2bdd0e5/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
--
diff --git a/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
b/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
index a89b0d7..77bbd16 100644
--- a/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
+++ b/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
@@ -19,6 +19,7 @@ package main.scala
import scala.util.Try
+import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext
git commit: Updated scripts for auditing releases
Repository: spark
Updated Branches:
refs/heads/branch-1.0 c3b406518 - 6541ca24b
Updated scripts for auditing releases
- Added script to automatically generate change list CHANGES.txt
- Added test for verifying linking against maven distributions of `spark-sql`
and `spark-hive`
- Added SBT projects for testing functionality of `spark-sql` and `spark-hive`
- Fixed issues in existing tests that might have come up because of changes in
Spark 1.0
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #844 from tdas/update-dev-scripts and squashes the following commits:
25090ba [Tathagata Das] Added missing license
e2e20b3 [Tathagata Das] Updated tests for auditing releases.
(cherry picked from commit b2bdd0e505f1ae3d39c46139f17bd43779ece635)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6541ca24
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6541ca24
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6541ca24
Branch: refs/heads/branch-1.0
Commit: 6541ca24b2cf79d6914b9d0bcf7b51bff405369e
Parents: c3b4065
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu May 22 20:48:55 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu May 22 20:49:25 2014 -0700
--
dev/audit-release/audit_release.py | 11 +-
.../maven_app_core/src/main/java/SimpleApp.java | 1 +
.../sbt_app_core/src/main/scala/SparkApp.scala | 2 +-
dev/audit-release/sbt_app_hive/build.sbt| 29 +++
dev/audit-release/sbt_app_hive/data.txt | 9 +
.../src/main/resources/hive-site.xml| 213 +++
.../sbt_app_hive/src/main/scala/HiveApp.scala | 57 +
dev/audit-release/sbt_app_sql/build.sbt | 29 +++
.../sbt_app_sql/src/main/scala/SqlApp.scala | 57 +
.../src/main/scala/StreamingApp.scala | 1 -
dev/create-release/generate-changelist.py | 144 +
11 files changed, 547 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6541ca24/dev/audit-release/audit_release.py
--
diff --git a/dev/audit-release/audit_release.py
b/dev/audit-release/audit_release.py
index 4a816d4..8c7573b 100755
--- a/dev/audit-release/audit_release.py
+++ b/dev/audit-release/audit_release.py
@@ -93,9 +93,12 @@ original_dir = os.getcwd()
# For each of these modules, we'll test an 'empty' application in sbt and
# maven that links against them. This will catch issues with messed up
# dependencies within those projects.
-modules = [spark-core, spark-bagel, spark-mllib, spark-streaming,
spark-repl,
- spark-graphx, spark-streaming-flume, spark-streaming-kafka,
- spark-streaming-mqtt, spark-streaming-twitter,
spark-streaming-zeromq]
+modules = [
+ spark-core, spark-bagel, spark-mllib, spark-streaming, spark-repl,
+ spark-graphx, spark-streaming-flume, spark-streaming-kafka,
+ spark-streaming-mqtt, spark-streaming-twitter, spark-streaming-zeromq,
+ spark-catalyst, spark-sql, spark-hive
+]
modules = map(lambda m: %s_%s % (m, SCALA_BINARY_VERSION), modules)
# Check for directories that might interfere with tests
@@ -122,7 +125,7 @@ for module in modules:
os.chdir(original_dir)
# SBT application tests
-for app in [sbt_app_core, sbt_app_graphx, sbt_app_streaming]:
+for app in [sbt_app_core, sbt_app_graphx, sbt_app_streaming,
sbt_app_sql, sbt_app_hive]:
os.chdir(app)
ret = run_cmd(sbt clean run, exit_on_failure=False)
test(ret == 0, sbt application (%s) % app)
http://git-wip-us.apache.org/repos/asf/spark/blob/6541ca24/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
--
diff --git a/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
b/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
index 6b65dda..5217689 100644
--- a/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
+++ b/dev/audit-release/maven_app_core/src/main/java/SimpleApp.java
@@ -37,5 +37,6 @@ public class SimpleApp {
System.exit(-1);
}
System.out.println(Test succeeded);
+ sc.stop();
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/6541ca24/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
--
diff --git a/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
b/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
index a89b0d7..77bbd16 100644
--- a/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
+++ b/dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
@@ -19,6 +19,7 @@ package main.scala
import
git commit: SPARK-1911: Emphasize that Spark jars should be built with Java 6.
Repository: spark Updated Branches: refs/heads/master 5081a0a9d - 75a032777 SPARK-1911: Emphasize that Spark jars should be built with Java 6. This commit requires the user to manually say yes when buiding Spark without Java 6. The prompt can be bypassed with a flag (e.g. if the user is scripting around make-distribution). Author: Patrick Wendell pwend...@gmail.com Closes #859 from pwendell/java6 and squashes the following commits: 4921133 [Patrick Wendell] Adding Pyspark Notice fee8c9e [Patrick Wendell] SPARK-1911: Emphasize that Spark jars should be built with Java 6. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/75a03277 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/75a03277 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/75a03277 Branch: refs/heads/master Commit: 75a03277704f8618a0f1c41aecfb1ebd24a8ac1a Parents: 5081a0a Author: Patrick Wendell pwend...@gmail.com Authored: Sat May 24 18:27:00 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Sat May 24 18:27:00 2014 -0700 -- make-distribution.sh | 52 --- 1 file changed, 31 insertions(+), 21 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/75a03277/make-distribution.sh -- diff --git a/make-distribution.sh b/make-distribution.sh index 1c89027..ae52b49 100755 --- a/make-distribution.sh +++ b/make-distribution.sh @@ -46,27 +46,6 @@ set -e FWDIR=$(cd `dirname $0`; pwd) DISTDIR=$FWDIR/dist -if [ -z $JAVA_HOME ]; then - echo Error: JAVA_HOME is not set, cannot proceed. - exit -1 -fi - -JAVA_CMD=$JAVA_HOME/bin/java -JAVA_VERSION=$($JAVA_CMD -version 21) -if ! [[ $JAVA_VERSION =~ 1.6 ]]; then - echo ***NOTE***: JAVA_HOME is not set to a JDK 6 installation. The resulting - echo distribution will not support Java 6. See SPARK-1703. - echo Output from 'java -version' was: - echo $JAVA_VERSION -fi - -VERSION=$(mvn help:evaluate -Dexpression=project.version 2/dev/null | grep -v INFO | tail -n 1) -if [ $? != 0 ]; then -echo -e You need Maven installed to build Spark. -echo -e Download Maven from https://maven.apache.org/; -exit -1; -fi - # Initialize defaults SPARK_HADOOP_VERSION=1.0.4 SPARK_YARN=false @@ -88,6 +67,9 @@ while (( $# )); do --with-hive) SPARK_HIVE=true ;; +--skip-java-test) + SKIP_JAVA_TEST=true + ;; --with-tachyon) SPARK_TACHYON=true ;; @@ -102,6 +84,34 @@ while (( $# )); do shift done +if [ -z $JAVA_HOME ]; then + echo Error: JAVA_HOME is not set, cannot proceed. + exit -1 +fi + +VERSION=$(mvn help:evaluate -Dexpression=project.version 2/dev/null | grep -v INFO | tail -n 1) +if [ $? != 0 ]; then +echo -e You need Maven installed to build Spark. +echo -e Download Maven from https://maven.apache.org/; +exit -1; +fi + +JAVA_CMD=$JAVA_HOME/bin/java +JAVA_VERSION=$($JAVA_CMD -version 21) +if [[ ! $JAVA_VERSION =~ 1.6 -z $SKIP_JAVA_TEST ]]; then + echo ***NOTE***: JAVA_HOME is not set to a JDK 6 installation. The resulting + echo distribution may not work well with PySpark and will not run + echo with Java 6 (See SPARK-1703 and SPARK-1911). + echo This test can be disabled by adding --skip-java-test. + echo Output from 'java -version' was: + echo $JAVA_VERSION + read -p Would you like to continue anyways? [y,n]: -r + if [[ ! $REPLY =~ ^[Yy]$ ]]; then +echo Okay, exiting. +exit 1 + fi +fi + if [ $NAME == none ]; then NAME=$SPARK_HADOOP_VERSION fi
git commit: Updated CHANGES.txt
Repository: spark Updated Branches: refs/heads/branch-1.0 217bd5627 - 840609272 Updated CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/84060927 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/84060927 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/84060927 Branch: refs/heads/branch-1.0 Commit: 84060927262159c4c847034a50ff8092f0a72a44 Parents: 217bd56 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Sun May 25 02:20:13 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Sun May 25 02:20:13 2014 + -- CHANGES.txt | 87 +++- 1 file changed, 86 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/84060927/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d21f0ac..b097339 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,91 @@ Spark Change Log Release 1.0.0 + SPARK-1911: Emphasize that Spark jars should be built with Java 6. + Patrick Wendell pwend...@gmail.com + 2014-05-24 18:27:00 -0700 + Commit: 217bd56, github.com/apache/spark/pull/859 + + [SPARK-1900 / 1918] PySpark on YARN is broken + Andrew Or andrewo...@gmail.com + 2014-05-24 18:01:49 -0700 + Commit: 12f5ecc, github.com/apache/spark/pull/853 + + Update LBFGSSuite.scala + baishuo(白硕) vc_j...@hotmail.com + 2014-05-23 13:02:40 -0700 + Commit: 9be103a, github.com/apache/spark/pull/815 + + Updated scripts for auditing releases + Tathagata Das tathagata.das1...@gmail.com + 2014-05-22 20:48:55 -0700 + Commit: 6541ca2, github.com/apache/spark/pull/844 + + [SPARK-1896] Respect spark.master (and --master) before MASTER in spark-shell + Andrew Or andrewo...@gmail.com + 2014-05-22 20:32:27 -0700 + Commit: c3b4065, github.com/apache/spark/pull/846 + + [SPARK-1897] Respect spark.jars (and --jars) in spark-shell + Andrew Or andrewo...@gmail.com + 2014-05-22 20:25:41 -0700 + Commit: 23cc40e, github.com/apache/spark/pull/849 + + Fix UISuite unit test that fails under Jenkins contention + Aaron Davidson aa...@databricks.com + 2014-05-22 15:11:05 -0700 + Commit: a566216, github.com/apache/spark/pull/857 + + [SPARK-1870] Make spark-submit --jars work in yarn-cluster mode. + Xiangrui Meng m...@databricks.com + 2014-05-22 01:52:50 -0700 + Commit: 79cd26c, github.com/apache/spark/pull/848 + + Configuration documentation updates + Reynold Xin r...@apache.org + 2014-05-21 18:49:12 -0700 + Commit: 75af8bd, github.com/apache/spark/pull/851 + + [SPARK-1889] [SQL] Apply splitConjunctivePredicates to join condition while finding join ke... + Takuya UESHIN ues...@happy-camper.st + 2014-05-21 15:37:47 -0700 + Commit: 6e7934e, github.com/apache/spark/pull/836 + + [SPARK-1519] Support minPartitions param of wholeTextFiles() in PySpark + Kan Zhang kzh...@apache.org + 2014-05-21 13:26:53 -0700 + Commit: 30d1df5, github.com/apache/spark/pull/697 + + [Typo] Stoped - Stopped + Andrew Or andrewo...@gmail.com + 2014-05-21 11:59:05 -0700 + Commit: 9b8f772, github.com/apache/spark/pull/847 + + [Minor] Move JdbcRDDSuite to the correct package + Andrew Or andrewo...@gmail.com + 2014-05-21 01:25:10 -0700 + Commit: bc6bbfa, github.com/apache/spark/pull/839 + + [Docs] Correct example of creating a new SparkConf + Andrew Or andrewo...@gmail.com + 2014-05-21 01:23:34 -0700 + Commit: 7295dd9, github.com/apache/spark/pull/842 + + [SPARK-1250] Fixed misleading comments in bin/pyspark, bin/spark-class + Sumedh Mungee smun...@gmail.com + 2014-05-21 01:22:25 -0700 + Commit: 364c14a, github.com/apache/spark/pull/843 + + [Hotfix] Blacklisted flaky HiveCompatibility test + Tathagata Das tathagata.das1...@gmail.com + 2014-05-20 10:27:12 -0700 + Commit: b4d93d3, github.com/apache/spark/pull/838 + + Updated CHANGES.txt + Tathagata Das tathagata.das1...@gmail.com + 2014-05-19 23:12:24 -0700 + Commit: 1c00f2a + [Spark 1877] ClassNotFoundException when loading RDD with serialized objects Tathagata Das tathagata.das1...@gmail.com, Ghidireac bogd...@u448a5b0a73d45358d94a.ant.amazon.com 2014-05-19 22:36:24 -0700 @@ -676,7 +761,7 @@ Release 1.0.0 Proposal: clarify Scala programming guide on caching ... Ethan Jewett esjew...@gmail.com 2014-05-06 20:50:08 -0700 - Commit: 51e27755, github.com/apache/spark/pull/668 + Commit: 51e2775, github.com/apache/spark/pull/668 SPARK-1727. Correct small compile errors, typos, and markdown issues in (primarly) MLlib docs Sean Owen so...@cloudera.com
git commit: [maven-release-plugin] prepare release v1.0.0-rc11
Repository: spark Updated Branches: refs/heads/branch-1.0 7273bfc03 - 2f1dc868e [maven-release-plugin] prepare release v1.0.0-rc11 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2f1dc868 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2f1dc868 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2f1dc868 Branch: refs/heads/branch-1.0 Commit: 2f1dc868e5714882cf40d2633fb66772baf34789 Parents: 7273bfc Author: Tathagata Das tathagata.das1...@gmail.com Authored: Sun May 25 10:18:41 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Sun May 25 10:18:41 2014 + -- assembly/pom.xml | 6 +++--- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 24 insertions(+), 24 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/2f1dc868/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 963357b..bed4261 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent @@ -122,8 +122,8 @@ transformer implementation=org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer resourcelog4j.properties/resource /transformer -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer/ -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer/ +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer / +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer / /transformers /configuration /execution http://git-wip-us.apache.org/repos/asf/spark/blob/2f1dc868/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 355f437..08932bb 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/2f1dc868/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index bab50f5..3e22641 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/2f1dc868/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 874bcd7..006757a 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/2f1dc868/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 6aec215..3ba984e 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../../pom.xml/relativePath /parent
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.0-rc11 [created] 522777f7e
[3/3] git commit: Updated CHANGES.txt
Updated CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/caed16e4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/caed16e4 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/caed16e4 Branch: refs/heads/branch-1.0 Commit: caed16e4c86427f13b3d4604c5dcb9ddcb71bbb1 Parents: 6d34a6a Author: Tathagata Das tathagata.das1...@gmail.com Authored: Mon May 26 06:16:25 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 26 06:16:25 2014 + -- CHANGES.txt | 65 1 file changed, 65 insertions(+) -- http://git-wip-us.apache.org/repos/asf/spark/blob/caed16e4/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index b097339..0cff06c 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,71 @@ Spark Change Log Release 1.0.0 + HOTFIX: Add no-arg SparkContext constructor in Java + Patrick Wendell pwend...@gmail.com + 2014-05-25 20:13:32 -0700 + Commit: 18c77cb, github.com/apache/spark/pull/878 + + [SQL] Minor: Introduce SchemaRDD#aggregate() for simple aggregations + Aaron Davidson aa...@databricks.com + 2014-05-25 18:37:44 -0700 + Commit: a3976a2, github.com/apache/spark/pull/874 + + SPARK-1903 Document Spark's network connections + Andrew Ash and...@andrewash.com + 2014-05-25 17:15:47 -0700 + Commit: 5107a6f, github.com/apache/spark/pull/856 + + Fix PEP8 violations in Python mllib. + Reynold Xin r...@apache.org + 2014-05-25 17:15:01 -0700 + Commit: 07f34ca, github.com/apache/spark/pull/871 + + Python docstring update for sql.py. + Reynold Xin r...@apache.org + 2014-05-25 16:04:17 -0700 + Commit: 8891495, github.com/apache/spark/pull/869 + + Fix PEP8 violations in examples/src/main/python. + Reynold Xin r...@apache.org + 2014-05-25 14:48:27 -0700 + Commit: 3368397, github.com/apache/spark/pull/870 + + Added license header for tox.ini. + Reynold Xin r...@apache.org + 2014-05-25 01:47:08 -0700 + Commit: 7273bfc + + SPARK-1822: Some minor cleanup work on SchemaRDD.count() + Reynold Xin r...@apache.org + 2014-05-25 01:44:49 -0700 + Commit: aeffc20, github.com/apache/spark/pull/868 + + Added PEP8 style configuration file. + Reynold Xin r...@apache.org + 2014-05-25 01:32:15 -0700 + Commit: 291567d, github.com/apache/spark/pull/872 + + [SPARK-1822] SchemaRDD.count() should use query optimizer + Kan Zhang kzh...@apache.org + 2014-05-25 00:06:42 -0700 + Commit: 64d0fb5, github.com/apache/spark/pull/841 + + spark-submit: add exec at the end of the script + Colin Patrick Mccabe cmcc...@cloudera.com + 2014-05-24 22:39:27 -0700 + Commit: 7e59335, github.com/apache/spark/pull/858 + + [SPARK-1886] check executor id existence when executor exit + Zhen Peng zhenpen...@baidu.com + 2014-05-24 20:40:19 -0700 + Commit: b5e9686, github.com/apache/spark/pull/827 + + Updated CHANGES.txt + Tathagata Das tathagata.das1...@gmail.com + 2014-05-25 02:20:13 + + Commit: 8406092 + SPARK-1911: Emphasize that Spark jars should be built with Java 6. Patrick Wendell pwend...@gmail.com 2014-05-24 18:27:00 -0700
[1/3] git commit: Revert [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.0 18c77cb53 - caed16e4c Revert [maven-release-plugin] prepare for next development iteration This reverts commit 832dc594e7666f1d402334f8015ce29917d9c888. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/73ffd1e5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/73ffd1e5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/73ffd1e5 Branch: refs/heads/branch-1.0 Commit: 73ffd1e547e14475c23e62402c800d7d18260365 Parents: 18c77cb Author: Tathagata Das tathagata.das1...@gmail.com Authored: Mon May 26 06:09:20 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 26 06:09:20 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/73ffd1e5/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 720ca77..bed4261 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/73ffd1e5/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 85f6d99..08932bb 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/73ffd1e5/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 47c2507..3e22641 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/73ffd1e5/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index b7cbb1a..006757a 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/73ffd1e5/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index b8fc07f..3ba984e 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/73ffd1e5/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 9eeb2e1..cb4dd47 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.1-SNAPSHOT/version +version1.0.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/73ffd1e5/external/mqtt/pom.xml -- diff --git
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.0-rc11 [deleted] 522777f7e
git commit: [maven-release-plugin] prepare release v1.0.0-rc11
Repository: spark Updated Branches: refs/heads/branch-1.0 caed16e4c - c69d97cdb [maven-release-plugin] prepare release v1.0.0-rc11 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c69d97cd Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c69d97cd Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c69d97cd Branch: refs/heads/branch-1.0 Commit: c69d97cdb42f809cb71113a1db4194c21372242a Parents: caed16e Author: Tathagata Das tathagata.das1...@gmail.com Authored: Mon May 26 06:46:48 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 26 06:46:48 2014 + -- assembly/pom.xml | 6 +++--- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 24 insertions(+), 24 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/c69d97cd/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 963357b..bed4261 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent @@ -122,8 +122,8 @@ transformer implementation=org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer resourcelog4j.properties/resource /transformer -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer/ -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer/ +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer / +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer / /transformers /configuration /execution http://git-wip-us.apache.org/repos/asf/spark/blob/c69d97cd/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 355f437..08932bb 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c69d97cd/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index bab50f5..3e22641 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c69d97cd/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 874bcd7..006757a 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c69d97cd/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 6aec215..3ba984e 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.0-SNAPSHOT/version +version1.0.0/version relativePath../../pom.xml/relativePath /parent
git commit: Streaming programming guide typos
Repository: spark
Updated Branches:
refs/heads/master d4c30cd99 - fdc4c112e
Streaming programming guide typos
Fix a bad Java code sample and a broken link in the streaming programming guide.
Author: Clément MATHIEU clem...@unportant.info
Closes #1286 from cykl/streaming-programming-guide-typos and squashes the
following commits:
b0908cb [Clément MATHIEU] Fix broken URL
9d3c535 [Clément MATHIEU] Spark streaming requires at least two working
threads (scala version was OK)
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fdc4c112
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fdc4c112
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fdc4c112
Branch: refs/heads/master
Commit: fdc4c112e7c2ac585d108d03209a642aa8bab7c8
Parents: d4c30cd
Author: Clément MATHIEU clem...@unportant.info
Authored: Thu Jul 3 18:31:18 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Jul 3 18:31:18 2014 -0700
--
docs/streaming-programming-guide.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/fdc4c112/docs/streaming-programming-guide.md
--
diff --git a/docs/streaming-programming-guide.md
b/docs/streaming-programming-guide.md
index ce8e58d..90a0eef 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -148,7 +148,7 @@ import org.apache.spark.streaming.*;
import org.apache.spark.streaming.api.java.*;
import scala.Tuple2;
// Create a StreamingContext with a local master
-JavaStreamingContext jssc = new JavaStreamingContext(local,
JavaNetworkWordCount, new Duration(1000))
+JavaStreamingContext jssc = new JavaStreamingContext(local[2],
JavaNetworkWordCount, new Duration(1000))
{% endhighlight %}
Using this context, we then create a new DStream
@@ -216,7 +216,7 @@ jssc.awaitTermination(); // Wait for the computation to
terminate
{% endhighlight %}
The complete code can be found in the Spark Streaming example
-[JavaNetworkWordCount]({{site.SPARK_GITHUB_URL}}/blob/master/examples/src/main/java/index.html?org/apache/spark/examples/streaming/JavaNetworkWordCount.java).
+[JavaNetworkWordCount]({{site.SPARK_GITHUB_URL}}/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java).
br
/div
git commit: [SPARK-2362] Fix for newFilesOnly logic in file DStream
Repository: spark Updated Branches: refs/heads/branch-1.0 1c12b0b5c - 885489112 [SPARK-2362] Fix for newFilesOnly logic in file DStream The newFilesOnly logic should be inverted: the logic should be that if the flag newFilesOnly==true then only start reading files older than current time. As the code is now if newFilesOnly==true then it will start to read files that are older than 0L (that is: every file in the directory). Author: Gabriele Nizzoli m...@nizzoli.net Closes #1077 from gabrielenizzoli/master and squashes the following commits: 4f1d261 [Gabriele Nizzoli] Fix for newFilesOnly logic in file DStream (cherry picked from commit e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912) Signed-off-by: Tathagata Das tathagata.das1...@gmail.com Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/88548911 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/88548911 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/88548911 Branch: refs/heads/branch-1.0 Commit: 885489112c82eb909df7efbf0515fd7abfae41a4 Parents: 1c12b0b Author: Gabriele Nizzoli m...@nizzoli.net Authored: Tue Jul 8 14:23:38 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Tue Jul 8 14:24:50 2014 -0700 -- .../org/apache/spark/streaming/dstream/FileInputDStream.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/88548911/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala -- diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala index e878285..9eecbfa 100644 --- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala +++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala @@ -45,7 +45,7 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F : NewInputFormat[K,V] : Clas // Files with mod time earlier than this is ignored. This is updated every interval // such that in the current interval, files older than any file found in the // previous interval will be ignored. Obviously this time keeps moving forward. - private var ignoreTime = if (newFilesOnly) 0L else System.currentTimeMillis() + private var ignoreTime = if (newFilesOnly) System.currentTimeMillis() else 0L // Latest file mod time seen till any point of time @transient private var path_ : Path = null
git commit: [STREAMING] SPARK-2343: Fix QueueInputDStream with oneAtATime false
Repository: spark
Updated Branches:
refs/heads/branch-1.0 d569838bc - 21fae6d3e
[STREAMING] SPARK-2343: Fix QueueInputDStream with oneAtATime false
Fix QueueInputDStream which was not removing dequeued items when used with the
oneAtATime flag disabled.
Author: Manuel Laflamme manuel.lafla...@gmail.com
Closes #1285 from mlaflamm/spark-2343 and squashes the following commits:
61c9e38 [Manuel Laflamme] Unit tests for queue input stream
c51d029 [Manuel Laflamme] Fix QueueInputDStream with oneAtATime false
(cherry picked from commit 0eb11527d13083ced215e3fda44ed849198a57cb)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/21fae6d3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/21fae6d3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/21fae6d3
Branch: refs/heads/branch-1.0
Commit: 21fae6d3e6f96d7272b55b8ee2ed2571c558b1fe
Parents: d569838
Author: Manuel Laflamme manuel.lafla...@gmail.com
Authored: Wed Jul 9 10:45:45 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Wed Jul 9 10:46:12 2014 -0700
--
.../streaming/dstream/QueueInputDStream.scala | 2 +-
.../spark/streaming/InputStreamsSuite.scala | 92 +++-
2 files changed, 92 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/21fae6d3/streaming/src/main/scala/org/apache/spark/streaming/dstream/QueueInputDStream.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/QueueInputDStream.scala
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/QueueInputDStream.scala
index 6376cff..ed7da6d 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/dstream/QueueInputDStream.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/dstream/QueueInputDStream.scala
@@ -41,7 +41,7 @@ class QueueInputDStream[T: ClassTag](
if (oneAtATime queue.size 0) {
buffer += queue.dequeue()
} else {
- buffer ++= queue
+ buffer ++= queue.dequeueAll(_ = true)
}
if (buffer.size 0) {
if (oneAtATime) {
http://git-wip-us.apache.org/repos/asf/spark/blob/21fae6d3/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
--
diff --git
a/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
b/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
index cd0aa4d..cc4a650 100644
---
a/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
+++
b/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
@@ -29,7 +29,7 @@ import java.nio.charset.Charset
import java.util.concurrent.{Executors, TimeUnit, ArrayBlockingQueue}
import java.util.concurrent.atomic.AtomicInteger
-import scala.collection.mutable.{SynchronizedBuffer, ArrayBuffer}
+import scala.collection.mutable.{SynchronizedBuffer, ArrayBuffer,
SynchronizedQueue}
import com.google.common.io.Files
import org.scalatest.BeforeAndAfter
@@ -39,6 +39,7 @@ import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.util.ManualClock
import org.apache.spark.util.Utils
import org.apache.spark.streaming.receiver.{ActorHelper, Receiver}
+import org.apache.spark.rdd.RDD
class InputStreamsSuite extends TestSuiteBase with BeforeAndAfter {
@@ -234,6 +235,95 @@ class InputStreamsSuite extends TestSuiteBase with
BeforeAndAfter {
logInfo()
assert(output.sum === numTotalRecords)
}
+
+ test(queue input stream - oneAtATime=true) {
+// Set up the streaming context and input streams
+val ssc = new StreamingContext(conf, batchDuration)
+val queue = new SynchronizedQueue[RDD[String]]()
+val queueStream = ssc.queueStream(queue, oneAtATime = true)
+val outputBuffer = new ArrayBuffer[Seq[String]] with
SynchronizedBuffer[Seq[String]]
+val outputStream = new TestOutputStream(queueStream, outputBuffer)
+def output = outputBuffer.filter(_.size 0)
+outputStream.register()
+ssc.start()
+
+// Setup data queued into the stream
+val clock = ssc.scheduler.clock.asInstanceOf[ManualClock]
+val input = Seq(1, 2, 3, 4, 5)
+val expectedOutput = input.map(Seq(_))
+//Thread.sleep(1000)
+val inputIterator = input.toIterator
+for (i - 0 until input.size) {
+ // Enqueue more than 1 item per tick but they should dequeue one at a
time
+ inputIterator.take(2).foreach(i = queue +=
ssc.sparkContext.makeRDD(Seq(i)))
+ clock.addToTime(batchDuration.milliseconds)
+}
+Thread.sleep(1000)
+logInfo(Stopping context)
+
git commit: [SPARK-1478].3: Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915
Repository: spark
Updated Branches:
refs/heads/master 369aa84e8 - 40a8fef4e
[SPARK-1478].3: Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915
This is a modified version of this PR https://github.com/apache/spark/pull/1168
done by @tmalaska
Adds MIMA binary check exclusions.
Author: tmalaska ted.mala...@cloudera.com
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #1347 from tdas/FLUME-1915 and squashes the following commits:
96065df [Tathagata Das] Added Mima exclusion for FlumeReceiver.
41d5338 [tmalaska] Address line 57 that was too long
12617e5 [tmalaska] SPARK-1478: Upgrade FlumeInputDStream's Flume...
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/40a8fef4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/40a8fef4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/40a8fef4
Branch: refs/heads/master
Commit: 40a8fef4e6619b4ea10a4ec9026260649ce5ae73
Parents: 369aa84
Author: tmalaska ted.mala...@cloudera.com
Authored: Thu Jul 10 13:15:02 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Jul 10 13:15:02 2014 -0700
--
.../streaming/flume/FlumeInputDStream.scala | 76 +---
.../spark/streaming/flume/FlumeUtils.scala | 41 ++-
.../streaming/flume/JavaFlumeStreamSuite.java | 2 +
.../streaming/flume/FlumeStreamSuite.scala | 41 +--
project/MimaExcludes.scala | 3 +
5 files changed, 147 insertions(+), 16 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/40a8fef4/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
--
diff --git
a/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
b/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
index ed35e34..07ae88f 100644
---
a/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
+++
b/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
@@ -20,6 +20,7 @@ package org.apache.spark.streaming.flume
import java.net.InetSocketAddress
import java.io.{ObjectInput, ObjectOutput, Externalizable}
import java.nio.ByteBuffer
+import java.util.concurrent.Executors
import scala.collection.JavaConversions._
import scala.reflect.ClassTag
@@ -29,24 +30,32 @@ import org.apache.flume.source.avro.AvroFlumeEvent
import org.apache.flume.source.avro.Status
import org.apache.avro.ipc.specific.SpecificResponder
import org.apache.avro.ipc.NettyServer
-
+import org.apache.spark.Logging
import org.apache.spark.util.Utils
import org.apache.spark.storage.StorageLevel
-import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.dstream._
-import org.apache.spark.Logging
+import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.receiver.Receiver
+import org.jboss.netty.channel.ChannelPipelineFactory
+import org.jboss.netty.channel.Channels
+import org.jboss.netty.channel.ChannelPipeline
+import org.jboss.netty.channel.ChannelFactory
+import org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory
+import org.jboss.netty.handler.codec.compression._
+import org.jboss.netty.handler.execution.ExecutionHandler
+
private[streaming]
class FlumeInputDStream[T: ClassTag](
@transient ssc_ : StreamingContext,
host: String,
port: Int,
- storageLevel: StorageLevel
+ storageLevel: StorageLevel,
+ enableDecompression: Boolean
) extends ReceiverInputDStream[SparkFlumeEvent](ssc_) {
override def getReceiver(): Receiver[SparkFlumeEvent] = {
-new FlumeReceiver(host, port, storageLevel)
+new FlumeReceiver(host, port, storageLevel, enableDecompression)
}
}
@@ -134,22 +143,71 @@ private[streaming]
class FlumeReceiver(
host: String,
port: Int,
-storageLevel: StorageLevel
+storageLevel: StorageLevel,
+enableDecompression: Boolean
) extends Receiver[SparkFlumeEvent](storageLevel) with Logging {
lazy val responder = new SpecificResponder(
classOf[AvroSourceProtocol], new FlumeEventServer(this))
- lazy val server = new NettyServer(responder, new InetSocketAddress(host,
port))
+ var server: NettyServer = null
+
+ private def initServer() = {
+if (enableDecompression) {
+ val channelFactory = new NioServerSocketChannelFactory
+(Executors.newCachedThreadPool(), Executors.newCachedThreadPool());
+ val channelPipelieFactory = new CompressionChannelPipelineFactory()
+
+ new NettyServer(
+responder,
+new InetSocketAddress(host, port),
+channelFactory,
+channelPipelieFactory,
+null)
+} else {
+ new
git commit: [SPARK-1341] [Streaming] Throttle BlockGenerator to limit rate of data consumption.
Repository: spark
Updated Branches:
refs/heads/master 40a8fef4e - 2dd672485
[SPARK-1341] [Streaming] Throttle BlockGenerator to limit rate of data
consumption.
Author: Issac Buenrostro buenros...@ooyala.com
Closes #945 from ibuenros/SPARK-1341-throttle and squashes the following
commits:
5514916 [Issac Buenrostro] Formatting changes, added documentation for
streaming throttling, stricter unit tests for throttling.
62f395f [Issac Buenrostro] Add comments and license to streaming
RateLimiter.scala
7066438 [Issac Buenrostro] Moved throttle code to RateLimiter class, smoother
pushing when throttling active
ccafe09 [Issac Buenrostro] Throttle BlockGenerator to limit rate of data
consumption.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2dd67248
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2dd67248
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2dd67248
Branch: refs/heads/master
Commit: 2dd67248503306bb08946b1796821e9f9ed4d00e
Parents: 40a8fef
Author: Issac Buenrostro buenros...@ooyala.com
Authored: Thu Jul 10 16:01:08 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Jul 10 16:01:08 2014 -0700
--
docs/configuration.md | 9 +++
.../streaming/receiver/BlockGenerator.scala | 3 +-
.../spark/streaming/receiver/RateLimiter.scala | 69
.../spark/streaming/NetworkReceiverSuite.scala | 38 +++
4 files changed, 118 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/2dd67248/docs/configuration.md
--
diff --git a/docs/configuration.md b/docs/configuration.md
index b84104c..0aea23a 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -774,6 +774,15 @@ Apart from these, the following properties are also
available, and may be useful
/td
/tr
tr
+ tdcodespark.streaming.receiver.maxRate/code/td
+ tdinfinite/td
+ td
+Maximum rate (per second) at which each receiver will push data into
blocks. Effectively,
+each stream will consume at most this number of records per second.
+Setting this configuration to 0 or a negative number will put no limit on
the rate.
+ /td
+/tr
+tr
tdcodespark.streaming.unpersist/code/td
tdtrue/td
td
http://git-wip-us.apache.org/repos/asf/spark/blob/2dd67248/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
index 78cc2da..0316b68 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
@@ -44,7 +44,7 @@ private[streaming] class BlockGenerator(
listener: BlockGeneratorListener,
receiverId: Int,
conf: SparkConf
- ) extends Logging {
+ ) extends RateLimiter(conf) with Logging {
private case class Block(id: StreamBlockId, buffer: ArrayBuffer[Any])
@@ -81,6 +81,7 @@ private[streaming] class BlockGenerator(
* will be periodically pushed into BlockManager.
*/
def += (data: Any): Unit = synchronized {
+waitToPush()
currentBuffer += data
}
http://git-wip-us.apache.org/repos/asf/spark/blob/2dd67248/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
new file mode 100644
index 000..e4f6ba6
--- /dev/null
+++
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
git commit: [SPARK-1341] [Streaming] Throttle BlockGenerator to limit rate of data consumption.
Repository: spark
Updated Branches:
refs/heads/branch-1.0 cb443cf6c - ca19cfbcd
[SPARK-1341] [Streaming] Throttle BlockGenerator to limit rate of data
consumption.
Author: Issac Buenrostro buenros...@ooyala.com
Closes #945 from ibuenros/SPARK-1341-throttle and squashes the following
commits:
5514916 [Issac Buenrostro] Formatting changes, added documentation for
streaming throttling, stricter unit tests for throttling.
62f395f [Issac Buenrostro] Add comments and license to streaming
RateLimiter.scala
7066438 [Issac Buenrostro] Moved throttle code to RateLimiter class, smoother
pushing when throttling active
ccafe09 [Issac Buenrostro] Throttle BlockGenerator to limit rate of data
consumption.
(cherry picked from commit 2dd67248503306bb08946b1796821e9f9ed4d00e)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ca19cfbc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ca19cfbc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ca19cfbc
Branch: refs/heads/branch-1.0
Commit: ca19cfbcd5cfac9ad731350dfeea14355aec87d6
Parents: cb443cf
Author: Issac Buenrostro buenros...@ooyala.com
Authored: Thu Jul 10 16:01:08 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Jul 10 16:01:28 2014 -0700
--
docs/configuration.md | 9 +++
.../streaming/receiver/BlockGenerator.scala | 3 +-
.../spark/streaming/receiver/RateLimiter.scala | 69
.../spark/streaming/NetworkReceiverSuite.scala | 38 +++
4 files changed, 118 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/ca19cfbc/docs/configuration.md
--
diff --git a/docs/configuration.md b/docs/configuration.md
index 71fafa5..fad0daf 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -774,6 +774,15 @@ Apart from these, the following properties are also
available, and may be useful
/td
/tr
tr
+ tdcodespark.streaming.receiver.maxRate/code/td
+ tdinfinite/td
+ td
+Maximum rate (per second) at which each receiver will push data into
blocks. Effectively,
+each stream will consume at most this number of records per second.
+Setting this configuration to 0 or a negative number will put no limit on
the rate.
+ /td
+/tr
+tr
tdcodespark.streaming.unpersist/code/td
tdtrue/td
td
http://git-wip-us.apache.org/repos/asf/spark/blob/ca19cfbc/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
index 78cc2da..0316b68 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
@@ -44,7 +44,7 @@ private[streaming] class BlockGenerator(
listener: BlockGeneratorListener,
receiverId: Int,
conf: SparkConf
- ) extends Logging {
+ ) extends RateLimiter(conf) with Logging {
private case class Block(id: StreamBlockId, buffer: ArrayBuffer[Any])
@@ -81,6 +81,7 @@ private[streaming] class BlockGenerator(
* will be periodically pushed into BlockManager.
*/
def += (data: Any): Unit = synchronized {
+waitToPush()
currentBuffer += data
}
http://git-wip-us.apache.org/repos/asf/spark/blob/ca19cfbc/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
new file mode 100644
index 000..e4f6ba6
--- /dev/null
+++
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
git commit: SPARK-1478.2 Fix incorrect NioServerSocketChannelFactory constructor call
Repository: spark
Updated Branches:
refs/heads/master d988d345d - 1fcd5dcdd
SPARK-1478.2 Fix incorrect NioServerSocketChannelFactory constructor call
The line break inadvertently means this was interpreted as a call to the no-arg
constructor. This doesn't exist in older Netty even. (Also fixed a val name
typo.)
Author: Sean Owen sro...@gmail.com
Closes #1466 from srowen/SPARK-1478.2 and squashes the following commits:
59c3501 [Sean Owen] Line break caused Scala to interpret
NioServerSocketChannelFactory constructor as the no-arg version, which is not
even present in some versions of Netty
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1fcd5dcd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1fcd5dcd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1fcd5dcd
Branch: refs/heads/master
Commit: 1fcd5dcdd8edb0e6989278c95e7f2c7d86c4efb2
Parents: d988d34
Author: Sean Owen sro...@gmail.com
Authored: Thu Jul 17 12:20:48 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Jul 17 12:20:48 2014 -0700
--
.../apache/spark/streaming/flume/FlumeInputDStream.scala | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1fcd5dcd/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
--
diff --git
a/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
b/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
index 07ae88f..56d2886 100644
---
a/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
+++
b/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala
@@ -153,15 +153,15 @@ class FlumeReceiver(
private def initServer() = {
if (enableDecompression) {
- val channelFactory = new NioServerSocketChannelFactory
-(Executors.newCachedThreadPool(), Executors.newCachedThreadPool());
- val channelPipelieFactory = new CompressionChannelPipelineFactory()
+ val channelFactory = new
NioServerSocketChannelFactory(Executors.newCachedThreadPool(),
+
Executors.newCachedThreadPool())
+ val channelPipelineFactory = new CompressionChannelPipelineFactory()
new NettyServer(
responder,
new InetSocketAddress(host, port),
-channelFactory,
-channelPipelieFactory,
+channelFactory,
+channelPipelineFactory,
null)
} else {
new NettyServer(responder, new InetSocketAddress(host, port))
git commit: [SPARK-2464][Streaming] Fixed Twitter stream stopping bug
Repository: spark
Updated Branches:
refs/heads/master fec641b84 - a45d5480f
[SPARK-2464][Streaming] Fixed Twitter stream stopping bug
Stopping the Twitter Receiver would call twitter4j's TwitterStream.shutdown,
which in turn causes an Exception to be thrown to the listener. This exception
caused the Receiver to be restarted. This patch check whether the receiver was
stopped or not, and accordingly restarts on exception.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #1577 from tdas/twitter-stop and squashes the following commits:
011b525 [Tathagata Das] Fixed Twitter stream stopping bug.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a45d5480
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a45d5480
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a45d5480
Branch: refs/heads/master
Commit: a45d5480f65d2e969fc7fbd8f358b1717fb99bef
Parents: fec641b
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu Jul 24 15:59:09 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Jul 24 15:59:09 2014 -0700
--
.../spark/streaming/twitter/TwitterInputDStream.scala | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a45d5480/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
--
diff --git
a/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
b/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
index 5ea2e55..4eacc47 100644
---
a/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
+++
b/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
@@ -63,7 +63,8 @@ class TwitterReceiver(
storageLevel: StorageLevel
) extends Receiver[Status](storageLevel) with Logging {
- private var twitterStream: TwitterStream = _
+ @volatile private var twitterStream: TwitterStream = _
+ @volatile private var stopped = false
def onStart() {
try {
@@ -78,7 +79,9 @@ class TwitterReceiver(
def onScrubGeo(l: Long, l1: Long) {}
def onStallWarning(stallWarning: StallWarning) {}
def onException(e: Exception) {
- restart(Error receiving tweets, e)
+ if (!stopped) {
+restart(Error receiving tweets, e)
+ }
}
})
@@ -91,12 +94,14 @@ class TwitterReceiver(
}
setTwitterStream(newTwitterStream)
logInfo(Twitter receiver started)
+ stopped = false
} catch {
case e: Exception = restart(Error starting Twitter stream, e)
}
}
def onStop() {
+stopped = true
setTwitterStream(null)
logInfo(Twitter receiver stopped)
}
git commit: [SPARK-2464][Streaming] Fixed Twitter stream stopping bug
Repository: spark
Updated Branches:
refs/heads/branch-1.0 91241592a - 53b4e0f95
[SPARK-2464][Streaming] Fixed Twitter stream stopping bug
Stopping the Twitter Receiver would call twitter4j's TwitterStream.shutdown,
which in turn causes an Exception to be thrown to the listener. This exception
caused the Receiver to be restarted. This patch check whether the receiver was
stopped or not, and accordingly restarts on exception.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #1577 from tdas/twitter-stop and squashes the following commits:
011b525 [Tathagata Das] Fixed Twitter stream stopping bug.
(cherry picked from commit a45d5480f65d2e969fc7fbd8f358b1717fb99bef)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/53b4e0f9
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/53b4e0f9
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/53b4e0f9
Branch: refs/heads/branch-1.0
Commit: 53b4e0f95750f371db14f8da4b5c4a1c38301710
Parents: 9124159
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Thu Jul 24 15:59:09 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Thu Jul 24 16:00:49 2014 -0700
--
.../spark/streaming/twitter/TwitterInputDStream.scala | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/53b4e0f9/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
--
diff --git
a/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
b/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
index 5ea2e55..4eacc47 100644
---
a/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
+++
b/external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
@@ -63,7 +63,8 @@ class TwitterReceiver(
storageLevel: StorageLevel
) extends Receiver[Status](storageLevel) with Logging {
- private var twitterStream: TwitterStream = _
+ @volatile private var twitterStream: TwitterStream = _
+ @volatile private var stopped = false
def onStart() {
try {
@@ -78,7 +79,9 @@ class TwitterReceiver(
def onScrubGeo(l: Long, l1: Long) {}
def onStallWarning(stallWarning: StallWarning) {}
def onException(e: Exception) {
- restart(Error receiving tweets, e)
+ if (!stopped) {
+restart(Error receiving tweets, e)
+ }
}
})
@@ -91,12 +94,14 @@ class TwitterReceiver(
}
setTwitterStream(newTwitterStream)
logInfo(Twitter receiver started)
+ stopped = false
} catch {
case e: Exception = restart(Error starting Twitter stream, e)
}
}
def onStop() {
+stopped = true
setTwitterStream(null)
logInfo(Twitter receiver stopped)
}
[3/3] git commit: Updating versions for 1.0.2 release.
Updating versions for 1.0.2 release.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/70109da2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/70109da2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/70109da2
Branch: refs/heads/branch-1.0
Commit: 70109da212343601d428252e9d298f6affa457f3
Parents: b1e1917
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Fri Jul 25 03:09:36 2014 +
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Fri Jul 25 03:09:36 2014 +
--
ec2/spark_ec2.py | 2 +-
project/SparkBuild.scala | 2 +-
project/project/SparkPluginBuild.scala | 2 +-
python/pyspark/shell.py| 2 +-
repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala | 2 +-
yarn/alpha/pom.xml | 2 +-
6 files changed, 6 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/70109da2/ec2/spark_ec2.py
--
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index 83fe920..348277a 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -164,7 +164,7 @@ def is_active(instance):
# Return correct versions of Spark and Shark, given the supplied Spark version
def get_spark_shark_version(opts):
spark_shark_map = {0.7.3: 0.7.1, 0.8.0: 0.8.0, 0.8.1: 0.8.1,
0.9.0: 0.9.0,
-0.9.1: 0.9.1, 1.0.0: 1.0.0, 1.0.1: 1.0.0}
+0.9.1: 0.9.1, 1.0.0: 1.0.0, 1.0.1: 1.0.0, 1.0.2: 1.0.0}
version = opts.spark_version.replace(v, )
if version not in spark_shark_map:
print stderr, Don't know about Spark version: %s % version
http://git-wip-us.apache.org/repos/asf/spark/blob/70109da2/project/SparkBuild.scala
--
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index e8edbf0..5b687e7 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -32,7 +32,7 @@ import scala.collection.JavaConversions._
// import com.jsuereth.pgp.sbtplugin.PgpKeys._
object SparkBuild extends Build {
- val SPARK_VERSION = 1.0.2-SNAPSHOT
+ val SPARK_VERSION = 1.0.2
val SPARK_VERSION_SHORT = SPARK_VERSION.replaceAll(-SNAPSHOT, )
// Hadoop version to build against. For example, 1.0.4 for Apache
releases, or
http://git-wip-us.apache.org/repos/asf/spark/blob/70109da2/project/project/SparkPluginBuild.scala
--
diff --git a/project/project/SparkPluginBuild.scala
b/project/project/SparkPluginBuild.scala
index aca93e3..722d26f 100644
--- a/project/project/SparkPluginBuild.scala
+++ b/project/project/SparkPluginBuild.scala
@@ -26,7 +26,7 @@ import sbt.Keys._
object SparkPluginDef extends Build {
lazy val root = Project(plugins, file(.)) dependsOn(sparkStyle)
lazy val sparkStyle = Project(spark-style, file(spark-style), settings =
styleSettings)
- val sparkVersion = 1.0.2-SNAPSHOT
+ val sparkVersion = 1.0.2
// There is actually no need to publish this artifact.
def styleSettings = Defaults.defaultSettings ++ Seq (
name := spark-style,
http://git-wip-us.apache.org/repos/asf/spark/blob/70109da2/python/pyspark/shell.py
--
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index 5fa8178..e6c3306 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -46,7 +46,7 @@ print(Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
- /__ / .__/\_,_/_/ /_/\_\ version 1.0.2-SNAPSHOT
+ /__ / .__/\_,_/_/ /_/\_\ version 1.0.2
/_/
)
print(Using Python version %s (%s, %s) % (
http://git-wip-us.apache.org/repos/asf/spark/blob/70109da2/repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala
--
diff --git a/repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala
b/repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala
index d7c05c5..9490195 100644
--- a/repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala
+++ b/repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala
@@ -26,7 +26,7 @@ trait SparkILoopInit {
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
- /___/ .__/\_,_/_/ /_/\_\ version 1.0.2-SNAPSHOT
+ /___/ .__/\_,_/_/ /_/\_\ version 1.0.2
/_/
)
import Properties._
http://git-wip-us.apache.org/repos/asf/spark/blob/70109da2/yarn/alpha/pom.xml
--
[1/3] git commit: Revert [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.0 53b4e0f95 - 70109da21 Revert [maven-release-plugin] prepare for next development iteration This reverts commit baf92a0f2119867b1be540085ebe9f1a1c411ae8. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d10455c6 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d10455c6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d10455c6 Branch: refs/heads/branch-1.0 Commit: d10455c6f1f25da165e9a0a7f1b1be62586b99c2 Parents: 53b4e0f Author: Ubuntu ubu...@ip-172-31-4-51.us-west-2.compute.internal Authored: Fri Jul 25 02:20:14 2014 + Committer: Ubuntu ubu...@ip-172-31-4-51.us-west-2.compute.internal Committed: Fri Jul 25 02:20:14 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/d10455c6/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index fbb993c..6735379 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d10455c6/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 0f7b0d7..8a38b43 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d10455c6/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 59110c2..23eea6e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d10455c6/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 08a9ddd..d158a75 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d10455c6/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index a486354..f5f0d54 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d10455c6/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index f68ad12..8bc5c03 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d10455c6/external/mqtt/pom.xml
git commit: [maven-release-plugin] prepare release v1.0.2-rc1
Repository: spark Updated Branches: refs/heads/branch-1.0 797c663ae - 919c87f26 [maven-release-plugin] prepare release v1.0.2-rc1 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/919c87f2 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/919c87f2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/919c87f2 Branch: refs/heads/branch-1.0 Commit: 919c87f26a2655bfd5ae03958915b6804367c1d6 Parents: 797c663 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 11:30:01 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 11:30:01 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/919c87f2/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index e290e79..6735379 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/919c87f2/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index c8ad40f..8a38b43 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/919c87f2/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 2302d7b..23eea6e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/919c87f2/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 9156a11..d158a75 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/919c87f2/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 1cefa15..f5f0d54 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/919c87f2/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index cc05e69..8bc5c03 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/919c87f2/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ab5afa2..ca022e0 100644 ---
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.2-rc1 [created] 86a5fc071
git commit: [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.0 919c87f26 - edbd02fc6 [maven-release-plugin] prepare for next development iteration Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/edbd02fc Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/edbd02fc Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/edbd02fc Branch: refs/heads/branch-1.0 Commit: edbd02fc6873676e080101d407916efb64bdf71a Parents: 919c87f Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 11:30:10 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 11:30:10 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/edbd02fc/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 6735379..fbb993c 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/edbd02fc/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 8a38b43..0f7b0d7 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/edbd02fc/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 23eea6e..59110c2 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/edbd02fc/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index d158a75..08a9ddd 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/edbd02fc/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index f5f0d54..a486354 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/edbd02fc/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 8bc5c03..f68ad12 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/edbd02fc/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ca022e0..b047b0f 100644 ---
[2/2] git commit: Revert [maven-release-plugin] prepare release v1.0.2-rc1
Revert [maven-release-plugin] prepare release v1.0.2-rc1 This reverts commit 919c87f26a2655bfd5ae03958915b6804367c1d6. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/01fc6d88 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/01fc6d88 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/01fc6d88 Branch: refs/heads/branch-1.0 Commit: 01fc6d885aab03e12d34b6823e5a63ce87013e26 Parents: d9ccf7f Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 18:18:06 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 18:18:06 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/01fc6d88/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 6735379..e290e79 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/01fc6d88/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 8a38b43..c8ad40f 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/01fc6d88/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 23eea6e..2302d7b 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/01fc6d88/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index d158a75..9156a11 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/01fc6d88/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index f5f0d54..1cefa15 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/01fc6d88/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 8bc5c03..cc05e69 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/01fc6d88/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ca022e0..ab5afa2 100644 --- a/external/mqtt/pom.xml
git commit: [maven-release-plugin] prepare release v1.0.2-rc1
Repository: spark Updated Branches: refs/heads/branch-1.0 01fc6d885 - 08f601328 [maven-release-plugin] prepare release v1.0.2-rc1 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/08f60132 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/08f60132 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/08f60132 Branch: refs/heads/branch-1.0 Commit: 08f601328ad9e7334ef7deb3a9fff1343a3c4f30 Parents: 01fc6d8 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 18:43:18 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 18:43:18 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/08f60132/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index e290e79..6735379 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/08f60132/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index c8ad40f..8a38b43 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/08f60132/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 2302d7b..23eea6e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/08f60132/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 9156a11..d158a75 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/08f60132/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 1cefa15..f5f0d54 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/08f60132/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index cc05e69..8bc5c03 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/08f60132/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ab5afa2..ca022e0 100644 ---
git commit: [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.0 08f601328 - 54df1b8c3 [maven-release-plugin] prepare for next development iteration Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/54df1b8c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/54df1b8c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/54df1b8c Branch: refs/heads/branch-1.0 Commit: 54df1b8c31fa2de5b04ee4a5563706b2664f34f3 Parents: 08f6013 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 18:43:25 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 18:43:25 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/54df1b8c/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 6735379..fbb993c 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/54df1b8c/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 8a38b43..0f7b0d7 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/54df1b8c/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 23eea6e..59110c2 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/54df1b8c/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index d158a75..08a9ddd 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/54df1b8c/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index f5f0d54..a486354 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/54df1b8c/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 8bc5c03..f68ad12 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/54df1b8c/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ca022e0..b047b0f 100644 ---
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.2-rc1 [created] 6053ff1be
git commit: Updated CHANGES.txt
Repository: spark Updated Branches: refs/heads/branch-1.0 54df1b8c3 - 76117baa7 Updated CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/76117baa Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/76117baa Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/76117baa Branch: refs/heads/branch-1.0 Commit: 76117baa7690deeaa765f672795be82ca51a891f Parents: 54df1b8 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 20:50:50 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 20:50:50 2014 + -- CHANGES.txt | 298 + dev/create-release/generate-changelist.py | 7 +- 2 files changed, 301 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/76117baa/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a4ec2f1..18a31e0 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,304 @@ Spark Change Log +Release 1.0.2 + + [SPARK-2529] Clean closures in foreach and foreachPartition. + Reynold Xin r...@apache.org + 2014-07-25 01:10:05 -0700 + Commit: 797c663, github.com/apache/spark/pull/1583 + + Updating versions for 1.0.2 release. + Tathagata Das tathagata.das1...@gmail.com + 2014-07-25 03:09:36 + + Commit: 70109da + + [SPARK-2464][Streaming] Fixed Twitter stream stopping bug + Tathagata Das tathagata.das1...@gmail.com + 2014-07-24 15:59:09 -0700 + Commit: 53b4e0f, github.com/apache/spark/pull/1577 + + [SPARK-2603][SQL] Remove unnecessary toMap and toList in converting Java collections to Scala collections JsonRDD.scala + Yin Huai h...@cse.ohio-state.edu + 2014-07-24 11:19:19 -0700 + Commit: 9124159, github.com/apache/spark/pull/1504 + + [SPARK-2658][SQL] Add rule for true = 1. + Michael Armbrust mich...@databricks.com + 2014-07-23 22:52:49 -0700 + Commit: 6b08046, github.com/apache/spark/pull/1556 + + [SPARK-2615] [SQL] Add Equal Sign == Support for HiveQl + Cheng Hao hao.ch...@intel.com + 2014-07-22 18:13:28 -0700 + Commit: c6421b6, github.com/apache/spark/pull/1522 + + [SPARK-2561][SQL] Fix apply schema + Michael Armbrust mich...@databricks.com + 2014-07-21 18:18:17 -0700 + Commit: 84bbfbd, github.com/apache/spark/pull/1470 + + [SPARK-2494] [PySpark] make hash of None consistant cross machines + Davies Liu davies@gmail.com + 2014-07-21 11:59:54 -0700 + Commit: cdcd467, github.com/apache/spark/pull/1371 + + Revert [SPARK-1199][REPL] Remove VALId and use the original import style for defined classes. + Patrick Wendell pwend...@gmail.com + 2014-07-21 11:53:57 -0700 + Commit: e0cc384 + + [SPARK-2598] RangePartitioner's binary search does not use the given Ordering + Reynold Xin r...@apache.org + 2014-07-20 11:06:06 -0700 + Commit: 480669f, github.com/apache/spark/pull/1500 + + [SPARK-2524] missing document about spark.deploy.retainedDrivers + lianhuiwang lianhuiwan...@gmail.com, Wang Lianhui lianhuiwan...@gmail.com, unknown administra...@taguswang-pc1.tencent.com + 2014-07-19 20:46:59 -0700 + Commit: 11670bf, github.com/apache/spark/pull/1443 + + Typo fix to the programming guide in the docs + Cesar Arevalo ce...@zephyrhealthinc.com + 2014-07-19 20:20:07 -0700 + Commit: a0624e8, github.com/apache/spark/pull/1495 + + [SPARK-2540] [SQL] Add HiveDecimal HiveVarchar support in unwrapping data + Cheng Hao hao.ch...@intel.com + 2014-07-18 16:38:11 -0500 + Commit: 7611840, github.com/apache/spark/pull/1436 + + Added t2 instance types + Basit Mustafa basitmustafa@computes-things-for-basit.local + 2014-07-18 12:23:47 -0700 + Commit: 284bf10, github.com/apache/spark/pull/1446 + + [SPARK-2570] [SQL] Fix the bug of ClassCastException + Cheng Hao hao.ch...@intel.com + 2014-07-17 23:25:01 -0700 + Commit: d35837a, github.com/apache/spark/pull/1475 + + [SPARK-2534] Avoid pulling in the entire RDD in various operators (branch-1.0 backport) + Reynold Xin r...@apache.org + 2014-07-17 16:33:30 -0700 + Commit: 26c428a, github.com/apache/spark/pull/1469 + + [SPARK-2412] CoalescedRDD throws exception with certain pref locs + Aaron Davidson aa...@databricks.com + 2014-07-17 01:01:14 -0700 + Commit: 3bb5d2f, github.com/apache/spark/pull/1337 + + [SPARK-2154] Schedule next Driver when one completes (standalone mode) + Aaron Davidson aa...@databricks.com + 2014-07-16 14:16:48 -0700 + Commit: 0b0b895, github.com/apache/spark/pull/1405 + + SPARK-1097: Do not introduce deadlock while fixing concurrency bug + Aaron Davidson aa...@databricks.com + 2014-07-16 14:10:17 -0700 + Commit: 91e7a71, github.com/apache/spark/pull/1409 + + [SPARK-2518][SQL] Fix foldability of Substring expression. + Takuya
[1/2] git commit: Revert [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.0 76117baa7 - 4be1dbd82 Revert [maven-release-plugin] prepare for next development iteration This reverts commit 54df1b8c31fa2de5b04ee4a5563706b2664f34f3. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/57b5531a Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/57b5531a Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/57b5531a Branch: refs/heads/branch-1.0 Commit: 57b5531a44ee0d0bd34007f2b5fc22dd733cb25a Parents: 76117ba Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 20:54:02 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 20:54:02 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/57b5531a/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index fbb993c..6735379 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/57b5531a/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 0f7b0d7..8a38b43 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/57b5531a/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 59110c2..23eea6e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/57b5531a/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 08a9ddd..d158a75 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/57b5531a/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index a486354..f5f0d54 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/57b5531a/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index f68ad12..8bc5c03 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.3-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/57b5531a/external/mqtt/pom.xml -- diff --git
[2/2] git commit: Revert [maven-release-plugin] prepare release v1.0.2-rc1
Revert [maven-release-plugin] prepare release v1.0.2-rc1 This reverts commit 08f601328ad9e7334ef7deb3a9fff1343a3c4f30. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4be1dbd8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4be1dbd8 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4be1dbd8 Branch: refs/heads/branch-1.0 Commit: 4be1dbd82302c0deb40f4f12f58129cc02a51842 Parents: 57b5531 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 20:54:14 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 20:54:14 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/4be1dbd8/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 6735379..e290e79 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4be1dbd8/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 8a38b43..c8ad40f 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4be1dbd8/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 23eea6e..2302d7b 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4be1dbd8/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index d158a75..9156a11 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4be1dbd8/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index f5f0d54..1cefa15 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4be1dbd8/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 8bc5c03..cc05e69 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.2-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4be1dbd8/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ca022e0..ab5afa2 100644 --- a/external/mqtt/pom.xml
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.2-rc1 [deleted] 6053ff1be
git commit: [maven-release-plugin] prepare release v1.0.2-rc1
Repository: spark Updated Branches: refs/heads/branch-1.0 4be1dbd82 - 8fb6f00e1 [maven-release-plugin] prepare release v1.0.2-rc1 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8fb6f00e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8fb6f00e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8fb6f00e Branch: refs/heads/branch-1.0 Commit: 8fb6f00e195fb258f3f70f04756e07c259a2351f Parents: 4be1dbd Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 21:21:15 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 21:21:15 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/8fb6f00e/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index e290e79..6735379 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8fb6f00e/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index c8ad40f..8a38b43 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8fb6f00e/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 2302d7b..23eea6e 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8fb6f00e/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 9156a11..d158a75 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8fb6f00e/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 1cefa15..f5f0d54 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8fb6f00e/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index cc05e69..8bc5c03 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2-SNAPSHOT/version +version1.0.2/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8fb6f00e/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ab5afa2..ca022e0 100644 ---
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.2-rc1 [created] 913e3de6d
git commit: [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.0 8fb6f00e1 - 03b684253 [maven-release-plugin] prepare for next development iteration Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/03b68425 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/03b68425 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/03b68425 Branch: refs/heads/branch-1.0 Commit: 03b684253d550ee199323b787bea68bacb782689 Parents: 8fb6f00 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Fri Jul 25 21:21:21 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Fri Jul 25 21:21:21 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 21 files changed, 22 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/03b68425/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 6735379..fbb993c 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/03b68425/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 8a38b43..0f7b0d7 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/03b68425/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 23eea6e..59110c2 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/03b68425/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index d158a75..08a9ddd 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/03b68425/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index f5f0d54..a486354 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/03b68425/external/kafka/pom.xml -- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 8bc5c03..f68ad12 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.0.2/version +version1.0.3-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/03b68425/external/mqtt/pom.xml -- diff --git a/external/mqtt/pom.xml b/external/mqtt/pom.xml index ca022e0..b047b0f 100644 ---
git commit: [SPARK-2103][Streaming] Change to ClassTag for KafkaInputDStream and fix reflection issue
Repository: spark
Updated Branches:
refs/heads/master 284771efb - a32f0fb73
[SPARK-2103][Streaming] Change to ClassTag for KafkaInputDStream and fix
reflection issue
This PR updates previous Manifest for KafkaInputDStream's Decoder to ClassTag,
also fix the problem addressed in
[SPARK-2103](https://issues.apache.org/jira/browse/SPARK-2103).
Previous Java interface cannot actually get the type of Decoder, so when using
this Manifest to reconstruct the decode object will meet reflection exception.
Also for other two Java interfaces, ClassTag[String] is useless because calling
Scala API will get the right implicit ClassTag.
Current Kafka unit test cannot actually verify the interface. I've tested these
interfaces in my local and distribute settings.
Author: jerryshao saisai.s...@intel.com
Closes #1508 from jerryshao/SPARK-2103 and squashes the following commits:
e90c37b [jerryshao] Add Mima excludes
7529810 [jerryshao] Change Manifest to ClassTag for KafkaInputDStream's Decoder
and fix Decoder construct issue when using Java API
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a32f0fb7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a32f0fb7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a32f0fb7
Branch: refs/heads/master
Commit: a32f0fb73a739c56208cafcd9f08618fb6dd8859
Parents: 284771e
Author: jerryshao saisai.s...@intel.com
Authored: Fri Aug 1 04:32:46 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Fri Aug 1 04:32:46 2014 -0700
--
.../spark/streaming/kafka/KafkaInputDStream.scala | 14 +++---
.../apache/spark/streaming/kafka/KafkaUtils.scala | 16 +---
project/MimaExcludes.scala | 7 ++-
3 files changed, 18 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a32f0fb7/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaInputDStream.scala
--
diff --git
a/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaInputDStream.scala
b/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaInputDStream.scala
index 38095e8..e20e2c8 100644
---
a/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaInputDStream.scala
+++
b/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaInputDStream.scala
@@ -18,7 +18,7 @@
package org.apache.spark.streaming.kafka
import scala.collection.Map
-import scala.reflect.ClassTag
+import scala.reflect.{classTag, ClassTag}
import java.util.Properties
import java.util.concurrent.Executors
@@ -48,8 +48,8 @@ private[streaming]
class KafkaInputDStream[
K: ClassTag,
V: ClassTag,
- U : Decoder[_]: Manifest,
- T : Decoder[_]: Manifest](
+ U : Decoder[_]: ClassTag,
+ T : Decoder[_]: ClassTag](
@transient ssc_ : StreamingContext,
kafkaParams: Map[String, String],
topics: Map[String, Int],
@@ -66,8 +66,8 @@ private[streaming]
class KafkaReceiver[
K: ClassTag,
V: ClassTag,
- U : Decoder[_]: Manifest,
- T : Decoder[_]: Manifest](
+ U : Decoder[_]: ClassTag,
+ T : Decoder[_]: ClassTag](
kafkaParams: Map[String, String],
topics: Map[String, Int],
storageLevel: StorageLevel
@@ -103,10 +103,10 @@ class KafkaReceiver[
tryZookeeperConsumerGroupCleanup(zkConnect, kafkaParams(group.id))
}
-val keyDecoder =
manifest[U].runtimeClass.getConstructor(classOf[VerifiableProperties])
+val keyDecoder =
classTag[U].runtimeClass.getConstructor(classOf[VerifiableProperties])
.newInstance(consumerConfig.props)
.asInstanceOf[Decoder[K]]
-val valueDecoder =
manifest[T].runtimeClass.getConstructor(classOf[VerifiableProperties])
+val valueDecoder =
classTag[T].runtimeClass.getConstructor(classOf[VerifiableProperties])
.newInstance(consumerConfig.props)
.asInstanceOf[Decoder[V]]
http://git-wip-us.apache.org/repos/asf/spark/blob/a32f0fb7/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
--
diff --git
a/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
b/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
index 86bb91f..48668f7 100644
---
a/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
+++
b/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
@@ -65,7 +65,7 @@ object KafkaUtils {
*in its own thread.
* @param storageLevel Storage level to use for storing the received objects
*/
- def createStream[K: ClassTag, V: ClassTag, U : Decoder[_]: Manifest, T :
git commit: [SPARK-2379] Fix the bug that streaming's receiver may fall into a dead loop
Repository: spark
Updated Branches:
refs/heads/master f5d9bea20 - b270309d7
[SPARK-2379] Fix the bug that streaming's receiver may fall into a dead loop
Author: joyyoj suns...@gmail.com
Closes #1694 from joyyoj/SPARK-2379 and squashes the following commits:
d73790d [joyyoj] SPARK-2379 Fix the bug that streaming's receiver may fall into
a dead loop
22e7821 [joyyoj] Merge remote-tracking branch 'apache/master'
3f4a602 [joyyoj] Merge remote-tracking branch 'remotes/apache/master'
f4660c5 [joyyoj] [SPARK-1998] SparkFlumeEvent with body bigger than 1020 bytes
are not read properly
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b270309d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b270309d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b270309d
Branch: refs/heads/master
Commit: b270309d7608fb749e402cd5afd36087446be398
Parents: f5d9bea
Author: joyyoj suns...@gmail.com
Authored: Fri Aug 1 13:41:55 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Fri Aug 1 13:41:55 2014 -0700
--
.../org/apache/spark/streaming/receiver/ReceiverSupervisor.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b270309d/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
index 09be3a5..1f0244c 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
@@ -138,7 +138,7 @@ private[streaming] abstract class ReceiverSupervisor(
onReceiverStop(message, error)
} catch {
case t: Throwable =
-stop(Error stopping receiver + streamId, Some(t))
+logError(Error stopping receiver + streamId + t.getStackTraceString)
}
}
git commit: [SPARK-2379] Fix the bug that streaming's receiver may fall into a dead loop
Repository: spark
Updated Branches:
refs/heads/branch-1.0 886508d3b - 952e0d698
[SPARK-2379] Fix the bug that streaming's receiver may fall into a dead loop
Author: joyyoj suns...@gmail.com
Closes #1694 from joyyoj/SPARK-2379 and squashes the following commits:
d73790d [joyyoj] SPARK-2379 Fix the bug that streaming's receiver may fall into
a dead loop
22e7821 [joyyoj] Merge remote-tracking branch 'apache/master'
3f4a602 [joyyoj] Merge remote-tracking branch 'remotes/apache/master'
f4660c5 [joyyoj] [SPARK-1998] SparkFlumeEvent with body bigger than 1020 bytes
are not read properly
(cherry picked from commit b270309d7608fb749e402cd5afd36087446be398)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/952e0d69
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/952e0d69
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/952e0d69
Branch: refs/heads/branch-1.0
Commit: 952e0d69841b6218e7d1b8b23e7d74a4fcb1b381
Parents: 886508d
Author: joyyoj suns...@gmail.com
Authored: Fri Aug 1 13:41:55 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Fri Aug 1 13:42:16 2014 -0700
--
.../org/apache/spark/streaming/receiver/ReceiverSupervisor.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/952e0d69/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
index 09be3a5..1f0244c 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
@@ -138,7 +138,7 @@ private[streaming] abstract class ReceiverSupervisor(
onReceiverStop(message, error)
} catch {
case t: Throwable =
-stop(Error stopping receiver + streamId, Some(t))
+logError(Error stopping receiver + streamId + t.getStackTraceString)
}
}
[1/2] [SPARK-1981] Add AWS Kinesis streaming support
Repository: spark
Updated Branches:
refs/heads/master 67bd8e3c2 - 91f9504e6
http://git-wip-us.apache.org/repos/asf/spark/blob/91f9504e/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
--
diff --git
a/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
b/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
new file mode 100644
index 000..41dbd64
--- /dev/null
+++
b/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.nio.ByteBuffer
+
+import scala.collection.JavaConversions.seqAsJavaList
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.Seconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.TestSuiteBase
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.ManualClock
+import org.scalatest.BeforeAndAfter
+import org.scalatest.Matchers
+import org.scalatest.mock.EasyMockSugar
+
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import
com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
+import
com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownReason
+import com.amazonaws.services.kinesis.model.Record
+
+/**
+ * Suite of Kinesis streaming receiver tests focusing mostly on the
KinesisRecordProcessor
+ */
+class KinesisReceiverSuite extends TestSuiteBase with Matchers with
BeforeAndAfter
+with EasyMockSugar {
+
+ val app = TestKinesisReceiver
+ val stream = mySparkStream
+ val endpoint = endpoint-url
+ val workerId = dummyWorkerId
+ val shardId = dummyShardId
+
+ val record1 = new Record()
+ record1.setData(ByteBuffer.wrap(Spark In Action.getBytes()))
+ val record2 = new Record()
+ record2.setData(ByteBuffer.wrap(Learning Spark.getBytes()))
+ val batch = List[Record](record1, record2)
+
+ var receiverMock: KinesisReceiver = _
+ var checkpointerMock: IRecordProcessorCheckpointer = _
+ var checkpointClockMock: ManualClock = _
+ var checkpointStateMock: KinesisCheckpointState = _
+ var currentClockMock: Clock = _
+
+ override def beforeFunction() = {
+receiverMock = mock[KinesisReceiver]
+checkpointerMock = mock[IRecordProcessorCheckpointer]
+checkpointClockMock = mock[ManualClock]
+checkpointStateMock = mock[KinesisCheckpointState]
+currentClockMock = mock[Clock]
+ }
+
+ test(kinesis utils api) {
+val ssc = new StreamingContext(master, framework, batchDuration)
+// Tests the API, does not actually test data receiving
+val kinesisStream = KinesisUtils.createStream(ssc, mySparkStream,
+ https://kinesis.us-west-2.amazonaws.com;, Seconds(2),
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2);
+ssc.stop()
+ }
+
+ test(process records including store and checkpoint) {
+val expectedCheckpointIntervalMillis = 10
+expecting {
+ receiverMock.isStopped().andReturn(false).once()
+ receiverMock.store(record1.getData().array()).once()
+ receiverMock.store(record2.getData().array()).once()
+ checkpointStateMock.shouldCheckpoint().andReturn(true).once()
+ checkpointerMock.checkpoint().once()
+ checkpointStateMock.advanceCheckpoint().once()
+}
+whenExecuting(receiverMock, checkpointerMock, checkpointStateMock) {
+ val recordProcessor = new KinesisRecordProcessor(receiverMock, workerId,
+ checkpointStateMock)
+
[2/2] git commit: [SPARK-1981] Add AWS Kinesis streaming support
[SPARK-1981] Add AWS Kinesis streaming support Author: Chris Fregly ch...@fregly.com Closes #1434 from cfregly/master and squashes the following commits: 4774581 [Chris Fregly] updated docs, renamed retry to retryRandom to be more clear, removed retries around store() method 0393795 [Chris Fregly] moved Kinesis examples out of examples/ and back into extras/kinesis-asl 691a6be [Chris Fregly] fixed tests and formatting, fixed a bug with JavaKinesisWordCount during union of streams 0e1c67b [Chris Fregly] Merge remote-tracking branch 'upstream/master' 74e5c7c [Chris Fregly] updated per TD's feedback. simplified examples, updated docs e33cbeb [Chris Fregly] Merge remote-tracking branch 'upstream/master' bf614e9 [Chris Fregly] per matei's feedback: moved the kinesis examples into the examples/ dir d17ca6d [Chris Fregly] per TD's feedback: updated docs, simplified the KinesisUtils api 912640c [Chris Fregly] changed the foundKinesis class to be a publically-avail class db3eefd [Chris Fregly] Merge remote-tracking branch 'upstream/master' 21de67f [Chris Fregly] Merge remote-tracking branch 'upstream/master' 6c39561 [Chris Fregly] parameterized the versions of the aws java sdk and kinesis client 338997e [Chris Fregly] improve build docs for kinesis 828f8ae [Chris Fregly] more cleanup e7c8978 [Chris Fregly] Merge remote-tracking branch 'upstream/master' cd68c0d [Chris Fregly] fixed typos and backward compatibility d18e680 [Chris Fregly] Merge remote-tracking branch 'upstream/master' b3b0ff1 [Chris Fregly] [SPARK-1981] Add AWS Kinesis streaming support Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/91f9504e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/91f9504e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/91f9504e Branch: refs/heads/master Commit: 91f9504e6086fac05b40545099f9818949c24bca Parents: 67bd8e3 Author: Chris Fregly ch...@fregly.com Authored: Sat Aug 2 13:35:35 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Sat Aug 2 13:35:35 2014 -0700 -- bin/run-example | 3 +- bin/run-example2.cmd| 3 +- dev/audit-release/audit_release.py | 4 +- .../sbt_app_core/src/main/scala/SparkApp.scala | 7 + dev/audit-release/sbt_app_kinesis/build.sbt | 28 ++ .../src/main/scala/SparkApp.scala | 33 +++ dev/create-release/create-release.sh| 4 +- dev/run-tests | 3 + docs/streaming-custom-receivers.md | 4 +- docs/streaming-kinesis.md | 58 docs/streaming-programming-guide.md | 12 +- examples/pom.xml| 13 + extras/kinesis-asl/pom.xml | 96 +++ .../streaming/JavaKinesisWordCountASL.java | 180 .../src/main/resources/log4j.properties | 37 +++ .../streaming/KinesisWordCountASL.scala | 251 + .../kinesis/KinesisCheckpointState.scala| 56 .../streaming/kinesis/KinesisReceiver.scala | 149 ++ .../kinesis/KinesisRecordProcessor.scala| 212 ++ .../spark/streaming/kinesis/KinesisUtils.scala | 96 +++ .../kinesis/JavaKinesisStreamSuite.java | 41 +++ .../src/test/resources/log4j.properties | 26 ++ .../kinesis/KinesisReceiverSuite.scala | 275 +++ pom.xml | 10 + project/SparkBuild.scala| 6 +- 25 files changed, 1592 insertions(+), 15 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/91f9504e/bin/run-example -- diff --git a/bin/run-example b/bin/run-example index 942706d..68a3570 100755 --- a/bin/run-example +++ b/bin/run-example @@ -29,7 +29,8 @@ if [ -n $1 ]; then else echo Usage: ./bin/run-example example-class [example-args] 12 echo - set MASTER=XX to use a specific master 12 - echo - can use abbreviated example class name (e.g. SparkPi, mllib.LinearRegression) 12 + echo - can use abbreviated example class name relative to com.apache.spark.examples 12 + echo (e.g. SparkPi, mllib.LinearRegression, streaming.KinesisWordCountASL) 12 exit 1 fi http://git-wip-us.apache.org/repos/asf/spark/blob/91f9504e/bin/run-example2.cmd -- diff --git a/bin/run-example2.cmd b/bin/run-example2.cmd index eadedd7..b29bf90 100644 --- a/bin/run-example2.cmd +++ b/bin/run-example2.cmd @@ -32,7 +32,8 @@ rem Test that an argument was given if not x%1==x goto arg_given echo Usage: run-example ^example-class^ [example-args] echo -
[2/2] git commit: [SPARK-1981] Add AWS Kinesis streaming support
[SPARK-1981] Add AWS Kinesis streaming support Author: Chris Fregly ch...@fregly.com Closes #1434 from cfregly/master and squashes the following commits: 4774581 [Chris Fregly] updated docs, renamed retry to retryRandom to be more clear, removed retries around store() method 0393795 [Chris Fregly] moved Kinesis examples out of examples/ and back into extras/kinesis-asl 691a6be [Chris Fregly] fixed tests and formatting, fixed a bug with JavaKinesisWordCount during union of streams 0e1c67b [Chris Fregly] Merge remote-tracking branch 'upstream/master' 74e5c7c [Chris Fregly] updated per TD's feedback. simplified examples, updated docs e33cbeb [Chris Fregly] Merge remote-tracking branch 'upstream/master' bf614e9 [Chris Fregly] per matei's feedback: moved the kinesis examples into the examples/ dir d17ca6d [Chris Fregly] per TD's feedback: updated docs, simplified the KinesisUtils api 912640c [Chris Fregly] changed the foundKinesis class to be a publically-avail class db3eefd [Chris Fregly] Merge remote-tracking branch 'upstream/master' 21de67f [Chris Fregly] Merge remote-tracking branch 'upstream/master' 6c39561 [Chris Fregly] parameterized the versions of the aws java sdk and kinesis client 338997e [Chris Fregly] improve build docs for kinesis 828f8ae [Chris Fregly] more cleanup e7c8978 [Chris Fregly] Merge remote-tracking branch 'upstream/master' cd68c0d [Chris Fregly] fixed typos and backward compatibility d18e680 [Chris Fregly] Merge remote-tracking branch 'upstream/master' b3b0ff1 [Chris Fregly] [SPARK-1981] Add AWS Kinesis streaming support (cherry picked from commit 91f9504e6086fac05b40545099f9818949c24bca) Signed-off-by: Tathagata Das tathagata.das1...@gmail.com Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bb0ac6d7 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bb0ac6d7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bb0ac6d7 Branch: refs/heads/branch-1.1 Commit: bb0ac6d7c91c491a99c252e6cb4aea40efe9b190 Parents: 91de0dc Author: Chris Fregly ch...@fregly.com Authored: Sat Aug 2 13:35:35 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Sat Aug 2 13:35:57 2014 -0700 -- bin/run-example | 3 +- bin/run-example2.cmd| 3 +- dev/audit-release/audit_release.py | 4 +- .../sbt_app_core/src/main/scala/SparkApp.scala | 7 + dev/audit-release/sbt_app_kinesis/build.sbt | 28 ++ .../src/main/scala/SparkApp.scala | 33 +++ dev/create-release/create-release.sh| 4 +- dev/run-tests | 3 + docs/streaming-custom-receivers.md | 4 +- docs/streaming-kinesis.md | 58 docs/streaming-programming-guide.md | 12 +- examples/pom.xml| 13 + extras/kinesis-asl/pom.xml | 96 +++ .../streaming/JavaKinesisWordCountASL.java | 180 .../src/main/resources/log4j.properties | 37 +++ .../streaming/KinesisWordCountASL.scala | 251 + .../kinesis/KinesisCheckpointState.scala| 56 .../streaming/kinesis/KinesisReceiver.scala | 149 ++ .../kinesis/KinesisRecordProcessor.scala| 212 ++ .../spark/streaming/kinesis/KinesisUtils.scala | 96 +++ .../kinesis/JavaKinesisStreamSuite.java | 41 +++ .../src/test/resources/log4j.properties | 26 ++ .../kinesis/KinesisReceiverSuite.scala | 275 +++ pom.xml | 10 + project/SparkBuild.scala| 6 +- 25 files changed, 1592 insertions(+), 15 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/bb0ac6d7/bin/run-example -- diff --git a/bin/run-example b/bin/run-example index 942706d..68a3570 100755 --- a/bin/run-example +++ b/bin/run-example @@ -29,7 +29,8 @@ if [ -n $1 ]; then else echo Usage: ./bin/run-example example-class [example-args] 12 echo - set MASTER=XX to use a specific master 12 - echo - can use abbreviated example class name (e.g. SparkPi, mllib.LinearRegression) 12 + echo - can use abbreviated example class name relative to com.apache.spark.examples 12 + echo (e.g. SparkPi, mllib.LinearRegression, streaming.KinesisWordCountASL) 12 exit 1 fi http://git-wip-us.apache.org/repos/asf/spark/blob/bb0ac6d7/bin/run-example2.cmd -- diff --git a/bin/run-example2.cmd b/bin/run-example2.cmd index eadedd7..b29bf90 100644 --- a/bin/run-example2.cmd +++ b/bin/run-example2.cmd @@ -32,7 +32,8 @@ rem
[1/2] [SPARK-1981] Add AWS Kinesis streaming support
Repository: spark
Updated Branches:
refs/heads/branch-1.1 91de0dc16 - bb0ac6d7c
http://git-wip-us.apache.org/repos/asf/spark/blob/bb0ac6d7/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
--
diff --git
a/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
b/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
new file mode 100644
index 000..41dbd64
--- /dev/null
+++
b/extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.kinesis
+
+import java.nio.ByteBuffer
+
+import scala.collection.JavaConversions.seqAsJavaList
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.Milliseconds
+import org.apache.spark.streaming.Seconds
+import org.apache.spark.streaming.StreamingContext
+import org.apache.spark.streaming.TestSuiteBase
+import org.apache.spark.streaming.util.Clock
+import org.apache.spark.streaming.util.ManualClock
+import org.scalatest.BeforeAndAfter
+import org.scalatest.Matchers
+import org.scalatest.mock.EasyMockSugar
+
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.InvalidStateException
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.KinesisClientLibDependencyException
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException
+import
com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException
+import
com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
+import
com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream
+import com.amazonaws.services.kinesis.clientlibrary.types.ShutdownReason
+import com.amazonaws.services.kinesis.model.Record
+
+/**
+ * Suite of Kinesis streaming receiver tests focusing mostly on the
KinesisRecordProcessor
+ */
+class KinesisReceiverSuite extends TestSuiteBase with Matchers with
BeforeAndAfter
+with EasyMockSugar {
+
+ val app = TestKinesisReceiver
+ val stream = mySparkStream
+ val endpoint = endpoint-url
+ val workerId = dummyWorkerId
+ val shardId = dummyShardId
+
+ val record1 = new Record()
+ record1.setData(ByteBuffer.wrap(Spark In Action.getBytes()))
+ val record2 = new Record()
+ record2.setData(ByteBuffer.wrap(Learning Spark.getBytes()))
+ val batch = List[Record](record1, record2)
+
+ var receiverMock: KinesisReceiver = _
+ var checkpointerMock: IRecordProcessorCheckpointer = _
+ var checkpointClockMock: ManualClock = _
+ var checkpointStateMock: KinesisCheckpointState = _
+ var currentClockMock: Clock = _
+
+ override def beforeFunction() = {
+receiverMock = mock[KinesisReceiver]
+checkpointerMock = mock[IRecordProcessorCheckpointer]
+checkpointClockMock = mock[ManualClock]
+checkpointStateMock = mock[KinesisCheckpointState]
+currentClockMock = mock[Clock]
+ }
+
+ test(kinesis utils api) {
+val ssc = new StreamingContext(master, framework, batchDuration)
+// Tests the API, does not actually test data receiving
+val kinesisStream = KinesisUtils.createStream(ssc, mySparkStream,
+ https://kinesis.us-west-2.amazonaws.com;, Seconds(2),
+ InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2);
+ssc.stop()
+ }
+
+ test(process records including store and checkpoint) {
+val expectedCheckpointIntervalMillis = 10
+expecting {
+ receiverMock.isStopped().andReturn(false).once()
+ receiverMock.store(record1.getData().array()).once()
+ receiverMock.store(record2.getData().array()).once()
+ checkpointStateMock.shouldCheckpoint().andReturn(true).once()
+ checkpointerMock.checkpoint().once()
+ checkpointStateMock.advanceCheckpoint().once()
+}
+whenExecuting(receiverMock, checkpointerMock, checkpointStateMock) {
+ val recordProcessor = new KinesisRecordProcessor(receiverMock, workerId,
+ checkpointStateMock)
+
svn commit: r6045 - /dev/spark/spark-1.0.2-rc1/
Author: tdas Date: Tue Aug 5 00:19:19 2014 New Revision: 6045 Log: Adding spark-1.0.2-rc1 Added: dev/spark/spark-1.0.2-rc1/ dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz (with props) dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.asc dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.md5 dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.sha dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz (with props) dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz.asc dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz.md5 dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz.sha dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop2.tgz (with props) dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop2.tgz.asc dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop2.tgz.md5 dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop2.tgz.sha dev/spark/spark-1.0.2-rc1/spark-1.0.2.tgz (with props) dev/spark/spark-1.0.2-rc1/spark-1.0.2.tgz.asc dev/spark/spark-1.0.2-rc1/spark-1.0.2.tgz.md5 dev/spark/spark-1.0.2-rc1/spark-1.0.2.tgz.sha Added: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz == Binary file - no diff available. Propchange: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz -- svn:mime-type = application/octet-stream Added: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.asc == --- dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.asc (added) +++ dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.asc Tue Aug 5 00:19:19 2014 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1.4.14 (GNU/Linux) + +iQIcBAABAgAGBQJT0tdUAAoJEF2VHP+H/RqXKDYQAJd5kz96LmBnpeGCx24JtSf2 +0vR4gTRZIYX2Er+PVY0GAKNjzTI8rij1aK7xGQmCsFvAmW0EYbXd7r5kyCqip+0d +Us7A8ultU9R1BPo2XaQgJSPg16xnT52ogSPXSOV40kLOyeWnQxiDL/O7/zThddjr +FHwMVvaS55dJOBzhqCPqFBV+19nqKgqIJ/j6PPvFhAIdI7NfMnb1cK0dyLeQEuG8 +uFTky8hNDbhYMskegKmShhsnN/Ub1DvYAAbinBdOIH1zJnyNjnyRjDYRUP+K0Hu5 +VThuQdeNHb7lwvktJgaxAjbsguAYJiRq+QA7yfj2QteB7E4ruGHZ8x48QcDKWYpv +4MDBfYRa4jjVX3c0rC0ZItfdErS54091z9sQIjIqTEk3pEr+AHOoptuBPEQKY3Ge +fKAJdqCViJtG+YU7vc5hfvQ92tg3kXAiHk0rAlat458Mag+NecJaCAWcDugrVxIw +V4qOWuih4nZRpRn1mZFnMXM6gkOxq2CZSwagX8hz7xRFrIO4SlgqYWuzX3M86OvS +ZAbaTL6loTOigA49AuW0ffBdgSf72Od0l9QSgu7Z9ufsiPeaqtUC5jXn6PLWw2gI +ZqtNSt2NNe7OOgXDcCOWjx1PcD+PKgI/99NJvhxeAv9H7zvsjqoV19eqsFqdP37W +6RMNUXNNeq5KnKqkS+Jk +=o9VB +-END PGP SIGNATURE- Added: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.md5 == --- dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.md5 (added) +++ dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.md5 Tue Aug 5 00:19:19 2014 @@ -0,0 +1 @@ +spark-1.0.2-bin-cdh4.tgz: DF C2 27 27 81 24 7A 24 1F 1F 58 78 44 3A 5F 19 Added: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.sha == --- dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.sha (added) +++ dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-cdh4.tgz.sha Tue Aug 5 00:19:19 2014 @@ -0,0 +1,3 @@ +spark-1.0.2-bin-cdh4.tgz: EC221342 C52DEBF6 5A52BBC9 6CCD1532 6FA4F177 37D09DD0 + E9F67B45 2174B721 F47ADE98 8F3CD8F0 008591F7 318C2856 + 0B3B224D D2E254CD F67790DC 0A136778 Added: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz == Binary file - no diff available. Propchange: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz -- svn:mime-type = application/octet-stream Added: dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz.asc == --- dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz.asc (added) +++ dev/spark/spark-1.0.2-rc1/spark-1.0.2-bin-hadoop1.tgz.asc Tue Aug 5 00:19:19 2014 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1.4.14 (GNU/Linux) + +iQIcBAABAgAGBQJT0tQsAAoJEF2VHP+H/RqXKKcP/34Q/wJqOonIyivWSHTsMl79 +EClGDar3jmJC0p3yPBdl6jZ+JVwkwTqlvHfeczM0POTGg0EeEcTJuk3wgB85FF6N +oHlKraQ4Rf0LEt1SmcrmtQLOOkrajWJWBH9VAXnGke9k2Kv9N5uu123l/b0jPONO +IVd09fdsBEK2lVYKi7Eyh3NQ4riDwPzk7ExDMHvSA+Ekp5V/zRSqgBSVOeziajAo +a2NhoJHaLHMhghL3zmqjJUH68Dyde8iYvJStrWSoHZJ0C+V14Lj0ASD/SA6wfnYO +KOKD1y2GTUlzoMpju5so90Dlap+0Rh2hobbZ2jv1mry+NQAwTUULFVCascBiq0mB +9HAaRmMRR/I4HPXk0JoTYHXdl6cgFBDms8sSKLEB2GLw06AqmhIUSUJ+hRvAPCqV +MJGsqzjn6wU9YvcinMjmdbPPBHYg3UdB/nT5F9TesbLSw0d7rEfpjMr2smpcM3Cw +klwhlp3o7R2mEkdLD2P9auBYbfY3Z7FEzPKQXp118i5CuTmfa4Kaq0bbhk8Iv8sH +GbaoKVFcJt3z95OTrX5N4Hy1Rn9xkwYaj5gal08OmUWsfuRxF9tb2+E5iu0uw4r0
Git Push Summary
Repository: spark Updated Tags: refs/tags/v0.7.0-bizo-1 [created] 02b4a0d96 refs/tags/v0.9.1-rc1 [created] 20a2d247a refs/tags/v0.9.1-rc2 [created] 95d70498b refs/tags/v0.9.1-rc3 [created] 931ac8a0e refs/tags/v1.0.0-rc1 [created] c8d0eb980 refs/tags/v1.0.0-rc10 [created] eafc6f6f4 refs/tags/v1.0.0-rc2 [created] 327ab1e24 refs/tags/v1.0.0-rc3 [created] e7c46933a refs/tags/v1.0.0-rc5 [created] 04ce12153 refs/tags/v1.0.0-rc6 [created] aab03f5f9 refs/tags/v1.0.0-rc7 [created] bc81ecfdc refs/tags/v1.0.0-rc8 [created] 9abe0192a refs/tags/v1.0.0-rc9 [created] 093aeb40f refs/tags/v1.0.1-rc1 [created] 2a9114542 refs/tags/v1.0.1-rc2 [created] 9c4493512 refs/tags/v1.0.2 [created] 8fb6f00e1 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.0-rc2 [deleted] 327ab1e24 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.0-rc7 [deleted] bc81ecfdc - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.0-rc6 [deleted] aab03f5f9 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.0-rc8 [deleted] 9abe0192a - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.1-rc1 [deleted] 2a9114542 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.0-rc10 [deleted] eafc6f6f4 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v0.9.2-rc1 [deleted] 5a3020615 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v0.9.1-rc3 [deleted] 931ac8a0e - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v0.9.1-rc1 [deleted] 20a2d247a - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v0.9.1-rc2 [deleted] 95d70498b - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.0.2-rc1 [deleted] 913e3de6d - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
git commit: Updating version for Spark 1.0.3 development.
Repository: spark
Updated Branches:
refs/heads/branch-1.0 439cdc581 - 3fbe3447e
Updating version for Spark 1.0.3 development.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3fbe3447
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3fbe3447
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3fbe3447
Branch: refs/heads/branch-1.0
Commit: 3fbe3447ec81b753985753dd47ad9941c78965e7
Parents: 439cdc5
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Mon Aug 4 18:45:18 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon Aug 4 18:45:18 2014 -0700
--
core/src/main/scala/org/apache/spark/SparkContext.scala| 2 +-
docs/_config.yml | 4 ++--
extras/java8-tests/pom.xml | 2 +-
project/SparkBuild.scala | 2 +-
project/project/SparkPluginBuild.scala | 2 +-
python/epydoc.conf | 2 +-
python/pyspark/shell.py| 2 +-
repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala | 2 +-
yarn/alpha/pom.xml | 2 +-
9 files changed, 10 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/3fbe3447/core/src/main/scala/org/apache/spark/SparkContext.scala
--
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 113c691..4a7d33d 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1265,7 +1265,7 @@ class SparkContext(config: SparkConf) extends Logging {
*/
object SparkContext extends Logging {
- private[spark] val SPARK_VERSION = 1.0.2
+ private[spark] val SPARK_VERSION = 1.0.3-SNAPSHOT
private[spark] val SPARK_JOB_DESCRIPTION = spark.job.description
http://git-wip-us.apache.org/repos/asf/spark/blob/3fbe3447/docs/_config.yml
--
diff --git a/docs/_config.yml b/docs/_config.yml
index 491b935..05a04fd 100644
--- a/docs/_config.yml
+++ b/docs/_config.yml
@@ -3,8 +3,8 @@ markdown: kramdown
# These allow the documentation to be updated with nerw releases
# of Spark, Scala, and Mesos.
-SPARK_VERSION: 1.0.2
-SPARK_VERSION_SHORT: 1.0.2
+SPARK_VERSION: 1.0.3
+SPARK_VERSION_SHORT: 1.0.3
SCALA_BINARY_VERSION: 2.10
SCALA_VERSION: 2.10.4
MESOS_VERSION: 0.18.1
http://git-wip-us.apache.org/repos/asf/spark/blob/3fbe3447/extras/java8-tests/pom.xml
--
diff --git a/extras/java8-tests/pom.xml b/extras/java8-tests/pom.xml
index b4ad113..4944d3d 100644
--- a/extras/java8-tests/pom.xml
+++ b/extras/java8-tests/pom.xml
@@ -20,7 +20,7 @@
parent
groupIdorg.apache.spark/groupId
artifactIdspark-parent/artifactId
-version1.0.2-SNAPSHOT/version
+version1.0.3-SNAPSHOT/version
relativePath../../pom.xml/relativePath
/parent
http://git-wip-us.apache.org/repos/asf/spark/blob/3fbe3447/project/SparkBuild.scala
--
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 5b687e7..5328ad3 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -32,7 +32,7 @@ import scala.collection.JavaConversions._
// import com.jsuereth.pgp.sbtplugin.PgpKeys._
object SparkBuild extends Build {
- val SPARK_VERSION = 1.0.2
+ val SPARK_VERSION = 1.0.3-SNAPSHOT
val SPARK_VERSION_SHORT = SPARK_VERSION.replaceAll(-SNAPSHOT, )
// Hadoop version to build against. For example, 1.0.4 for Apache
releases, or
http://git-wip-us.apache.org/repos/asf/spark/blob/3fbe3447/project/project/SparkPluginBuild.scala
--
diff --git a/project/project/SparkPluginBuild.scala
b/project/project/SparkPluginBuild.scala
index 722d26f..707c9d0 100644
--- a/project/project/SparkPluginBuild.scala
+++ b/project/project/SparkPluginBuild.scala
@@ -26,7 +26,7 @@ import sbt.Keys._
object SparkPluginDef extends Build {
lazy val root = Project(plugins, file(.)) dependsOn(sparkStyle)
lazy val sparkStyle = Project(spark-style, file(spark-style), settings =
styleSettings)
- val sparkVersion = 1.0.2
+ val sparkVersion = 1.0.3-SNAPSHOT
// There is actually no need to publish this artifact.
def styleSettings = Defaults.defaultSettings ++ Seq (
name := spark-style,
svn commit: r6047 - /dev/spark/spark-1.0.2-rc1/ /release/spark/spark-1.0.2/
Author: tdas Date: Tue Aug 5 01:53:43 2014 New Revision: 6047 Log: Moving Spark 1.0.2 from dev to release. Added: release/spark/spark-1.0.2/ - copied from r6046, dev/spark/spark-1.0.2-rc1/ Removed: dev/spark/spark-1.0.2-rc1/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-1022][Streaming] Add Kafka real unit test
Repository: spark
Updated Branches:
refs/heads/master 184048f80 - e87075df9
[SPARK-1022][Streaming] Add Kafka real unit test
This PR is a updated version of (https://github.com/apache/spark/pull/557) to
actually test sending and receiving data through Kafka, and fix previous flaky
issues.
@tdas, would you mind reviewing this PR? Thanks a lot.
Author: jerryshao saisai.s...@intel.com
Closes #1751 from jerryshao/kafka-unit-test and squashes the following commits:
b6a505f [jerryshao] code refactor according to comments
5222330 [jerryshao] Change JavaKafkaStreamSuite to better test it
5525f10 [jerryshao] Fix flaky issue of Kafka real unit test
4559310 [jerryshao] Minor changes for Kafka unit test
860f649 [jerryshao] Minor style changes, and tests ignored due to flakiness
796d4ca [jerryshao] Add real Kafka streaming test
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e87075df
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e87075df
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e87075df
Branch: refs/heads/master
Commit: e87075df977a539e4a1684045a7bd66c36285174
Parents: 184048f
Author: jerryshao saisai.s...@intel.com
Authored: Tue Aug 5 10:40:28 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Tue Aug 5 10:40:28 2014 -0700
--
external/kafka/pom.xml | 6 +
.../streaming/kafka/JavaKafkaStreamSuite.java | 125 ++--
.../streaming/kafka/KafkaStreamSuite.scala | 197 +--
3 files changed, 293 insertions(+), 35 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e87075df/external/kafka/pom.xml
--
diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml
index daf0336..2aee999 100644
--- a/external/kafka/pom.xml
+++ b/external/kafka/pom.xml
@@ -71,6 +71,12 @@
/exclusions
/dependency
dependency
+ groupIdnet.sf.jopt-simple/groupId
+ artifactIdjopt-simple/artifactId
+ version3.2/version
+ scopetest/scope
+/dependency
+dependency
groupIdorg.scalatest/groupId
artifactIdscalatest_${scala.binary.version}/artifactId
scopetest/scope
http://git-wip-us.apache.org/repos/asf/spark/blob/e87075df/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
--
diff --git
a/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
b/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
index 9f8046b..0571454 100644
---
a/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
+++
b/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
@@ -17,31 +17,118 @@
package org.apache.spark.streaming.kafka;
+import java.io.Serializable;
import java.util.HashMap;
+import java.util.List;
+
+import scala.Predef;
+import scala.Tuple2;
+import scala.collection.JavaConverters;
+
+import junit.framework.Assert;
-import org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream;
-import org.junit.Test;
-import com.google.common.collect.Maps;
import kafka.serializer.StringDecoder;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.Function;
import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.LocalJavaStreamingContext;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+
+import org.junit.Test;
+import org.junit.After;
+import org.junit.Before;
+
+public class JavaKafkaStreamSuite extends LocalJavaStreamingContext implements
Serializable {
+ private transient KafkaStreamSuite testSuite = new KafkaStreamSuite();
+
+ @Before
+ @Override
+ public void setUp() {
+testSuite.beforeFunction();
+System.clearProperty(spark.driver.port);
+//System.setProperty(spark.streaming.clock,
org.apache.spark.streaming.util.SystemClock);
+ssc = new JavaStreamingContext(local[2], test, new Duration(1000));
+ }
+
+ @After
+ @Override
+ public void tearDown() {
+ssc.stop();
+ssc = null;
+System.clearProperty(spark.driver.port);
+testSuite.afterFunction();
+ }
-public class JavaKafkaStreamSuite extends LocalJavaStreamingContext {
@Test
- public void testKafkaStream() {
-HashMapString, Integer topics = Maps.newHashMap();
-
-// tests the API, does not actually test data receiving
-JavaPairReceiverInputDStreamString, String test1 =
-KafkaUtils.createStream(ssc
git commit: [SPARK-1022][Streaming] Add Kafka real unit test
Repository: spark
Updated Branches:
refs/heads/branch-1.1 075ba6781 - b92a45058
[SPARK-1022][Streaming] Add Kafka real unit test
This PR is a updated version of (https://github.com/apache/spark/pull/557) to
actually test sending and receiving data through Kafka, and fix previous flaky
issues.
@tdas, would you mind reviewing this PR? Thanks a lot.
Author: jerryshao saisai.s...@intel.com
Closes #1751 from jerryshao/kafka-unit-test and squashes the following commits:
b6a505f [jerryshao] code refactor according to comments
5222330 [jerryshao] Change JavaKafkaStreamSuite to better test it
5525f10 [jerryshao] Fix flaky issue of Kafka real unit test
4559310 [jerryshao] Minor changes for Kafka unit test
860f649 [jerryshao] Minor style changes, and tests ignored due to flakiness
796d4ca [jerryshao] Add real Kafka streaming test
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b92a4505
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b92a4505
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b92a4505
Branch: refs/heads/branch-1.1
Commit: b92a450583989470ff53b62c124d908ad661e29a
Parents: 075ba67
Author: jerryshao saisai.s...@intel.com
Authored: Tue Aug 5 10:40:28 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Tue Aug 5 10:43:35 2014 -0700
--
external/kafka/pom.xml | 6 +
.../streaming/kafka/JavaKafkaStreamSuite.java | 125 ++--
.../streaming/kafka/KafkaStreamSuite.scala | 197 +--
3 files changed, 293 insertions(+), 35 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b92a4505/external/kafka/pom.xml
--
diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml
index daf0336..2aee999 100644
--- a/external/kafka/pom.xml
+++ b/external/kafka/pom.xml
@@ -71,6 +71,12 @@
/exclusions
/dependency
dependency
+ groupIdnet.sf.jopt-simple/groupId
+ artifactIdjopt-simple/artifactId
+ version3.2/version
+ scopetest/scope
+/dependency
+dependency
groupIdorg.scalatest/groupId
artifactIdscalatest_${scala.binary.version}/artifactId
scopetest/scope
http://git-wip-us.apache.org/repos/asf/spark/blob/b92a4505/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
--
diff --git
a/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
b/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
index 9f8046b..0571454 100644
---
a/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
+++
b/external/kafka/src/test/java/org/apache/spark/streaming/kafka/JavaKafkaStreamSuite.java
@@ -17,31 +17,118 @@
package org.apache.spark.streaming.kafka;
+import java.io.Serializable;
import java.util.HashMap;
+import java.util.List;
+
+import scala.Predef;
+import scala.Tuple2;
+import scala.collection.JavaConverters;
+
+import junit.framework.Assert;
-import org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream;
-import org.junit.Test;
-import com.google.common.collect.Maps;
import kafka.serializer.StringDecoder;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.Function;
import org.apache.spark.storage.StorageLevel;
+import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.LocalJavaStreamingContext;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaPairDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+
+import org.junit.Test;
+import org.junit.After;
+import org.junit.Before;
+
+public class JavaKafkaStreamSuite extends LocalJavaStreamingContext implements
Serializable {
+ private transient KafkaStreamSuite testSuite = new KafkaStreamSuite();
+
+ @Before
+ @Override
+ public void setUp() {
+testSuite.beforeFunction();
+System.clearProperty(spark.driver.port);
+//System.setProperty(spark.streaming.clock,
org.apache.spark.streaming.util.SystemClock);
+ssc = new JavaStreamingContext(local[2], test, new Duration(1000));
+ }
+
+ @After
+ @Override
+ public void tearDown() {
+ssc.stop();
+ssc = null;
+System.clearProperty(spark.driver.port);
+testSuite.afterFunction();
+ }
-public class JavaKafkaStreamSuite extends LocalJavaStreamingContext {
@Test
- public void testKafkaStream() {
-HashMapString, Integer topics = Maps.newHashMap();
-
-// tests the API, does not actually test data receiving
-JavaPairReceiverInputDStreamString, String test1
svn commit: r1616049 - in /spark: ./ _layouts/ news/_posts/ releases/_posts/ site/ site/docs/ site/docs/1.0.2/ site/docs/1.0.2/api/ site/docs/1.0.2/api/java/ site/docs/1.0.2/api/java/org/ site/docs/1.
Author: tdas Date: Tue Aug 5 23:40:54 2014 New Revision: 1616049 URL: http://svn.apache.org/r1616049 Log: Adding Spark 1.0.2 [This commit notification would consist of 386 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r1616050 - in /spark: news/_posts/2014-08-05-spark-1-0-2-released.md site/news/index.html site/news/spark-1-0-2-released.html
Author: tdas
Date: Tue Aug 5 23:43:56 2014
New Revision: 1616050
URL: http://svn.apache.org/r1616050
Log:
Small update to Spark 1.0.2.
Modified:
spark/news/_posts/2014-08-05-spark-1-0-2-released.md
spark/site/news/index.html
spark/site/news/spark-1-0-2-released.html
Modified: spark/news/_posts/2014-08-05-spark-1-0-2-released.md
URL:
http://svn.apache.org/viewvc/spark/news/_posts/2014-08-05-spark-1-0-2-released.md?rev=1616050r1=1616049r2=1616050view=diff
==
--- spark/news/_posts/2014-08-05-spark-1-0-2-released.md (original)
+++ spark/news/_posts/2014-08-05-spark-1-0-2-released.md Tue Aug 5 23:43:56
2014
@@ -11,6 +11,6 @@ meta:
_edit_last: '4'
_wpas_done_all: '1'
---
-We are happy to announce the availability of a
href={{site.url}}releases/spark-release-1-0-2.html title=Spark Release
1.0.2Spark 1.0.2/a! This release includes contributions from 30 developers.
Spark 1.0.2 includes fixes across several areas of Spark, including the core
API, PySpark, and MLlib.
+We are happy to announce the availability of a
href={{site.url}}releases/spark-release-1-0-2.html title=Spark Release
1.0.2Spark 1.0.2/a! This release includes contributions from 30 developers.
Spark 1.0.2 includes fixes across several areas of Spark, including the core
API, Streaming, PySpark, and MLlib.
Visit the a href={{site.url}}releases/spark-release-1-0-2.html title=Spark
Release 1.0.2release notes/a to read about this release or a
href={{site.url}}downloads.htmldownload/a the release today.
Modified: spark/site/news/index.html
URL:
http://svn.apache.org/viewvc/spark/site/news/index.html?rev=1616050r1=1616049r2=1616050view=diff
==
--- spark/site/news/index.html (original)
+++ spark/site/news/index.html Tue Aug 5 23:43:56 2014
@@ -164,7 +164,7 @@
h3 class=entry-titlea href=/news/spark-1-0-2-released.htmlSpark
1.0.2 released/a/h3
div class=entry-dateAugust 5, 2014/div
/header
-div class=entry-contentpWe are happy to announce the availability of
a href=/releases/spark-release-1-0-2.html title=Spark Release 1.0.2Spark
1.0.2/a! This release includes contributions from 30 developers. Spark 1.0.2
includes fixes across several areas of Spark, including the core API, PySpark,
and MLlib./p
+div class=entry-contentpWe are happy to announce the availability of
a href=/releases/spark-release-1-0-2.html title=Spark Release 1.0.2Spark
1.0.2/a! This release includes contributions from 30 developers. Spark 1.0.2
includes fixes across several areas of Spark, including the core API,
Streaming, PySpark, and MLlib./p
/div
/article
Modified: spark/site/news/spark-1-0-2-released.html
URL:
http://svn.apache.org/viewvc/spark/site/news/spark-1-0-2-released.html?rev=1616050r1=1616049r2=1616050view=diff
==
--- spark/site/news/spark-1-0-2-released.html (original)
+++ spark/site/news/spark-1-0-2-released.html Tue Aug 5 23:43:56 2014
@@ -160,7 +160,7 @@
h2Spark 1.0.2 released/h2
-pWe are happy to announce the availability of a
href=/releases/spark-release-1-0-2.html title=Spark Release 1.0.2Spark
1.0.2/a! This release includes contributions from 30 developers. Spark 1.0.2
includes fixes across several areas of Spark, including the core API, PySpark,
and MLlib./p
+pWe are happy to announce the availability of a
href=/releases/spark-release-1-0-2.html title=Spark Release 1.0.2Spark
1.0.2/a! This release includes contributions from 30 developers. Spark 1.0.2
includes fixes across several areas of Spark, including the core API,
Streaming, PySpark, and MLlib./p
pVisit the a href=/releases/spark-release-1-0-2.html title=Spark Release
1.0.2release notes/a to read about this release or a
href=/downloads.htmldownload/a the release today./p
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
