[GitHub] spark pull request: Make deprecation warning less severe

[pwendell]

GitHub user pwendell opened a pull request:

https://github.com/apache/spark/pull/810

Make deprecation warning less severe

Just a small change. I think it's good not to scare people who are using 
the old options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pwendell/spark warnings

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/810.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #810


commit cb8a3116f2b6a22af6d0f8a0aa9614c10bde0fcf
Author: Patrick Wendell 
Date:   2014-05-17T05:49:34Z

Make deprecation warning less severe




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Make deprecation warning less severe

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/810#issuecomment-43398666
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Make deprecation warning less severe

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/810


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/802


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12766116
  
--- Diff: bin/pyspark ---
@@ -25,6 +25,12 @@ export SPARK_HOME="$FWDIR"
 
 SCALA_VERSION=2.10
 
+if [[ "$@" = *--help ]] || [[ "$@" = *--h ]]; then
--- End diff --

I think maybe you want this to say 

```
if [[ "$@" = *--help ]] || [[ "$@" = *-h ]]; then
```

One thing is this will only detect if `-h` or `--help` is the 
_last_argument, but I think anything other than that is pretty tricky.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43398339
  
LGTM - thanks Andrew!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43398162
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/799


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43398156
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12765913
  
--- Diff: bin/pyspark ---
@@ -25,6 +25,12 @@ export SPARK_HOME="$FWDIR"
 
 SCALA_VERSION=2.10
 
+if [[ "$@" = *--help ]] || [[ "$@" = *--h ]]; then
--- End diff --

this doesn't seem to match `-h` for me... not sure if that was the 
intention of the second condition.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Version bump of spark-ec2 scripts

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/809


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43396764
  
LGTM - a few minor comments. Mind taking a quick look


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Version bump of spark-ec2 scripts

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/809#issuecomment-43397526
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15064/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Version bump of spark-ec2 scripts

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/809#issuecomment-43396873
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1817] RDD.zip() should verify partition...

[kanzhang]

Github user kanzhang commented on the pull request:

https://github.com/apache/spark/pull/760#issuecomment-43395929
  
@witgo yes, the first one is wrong and that is due to the Scala bug I 
referenced above (https://issues.scala-lang.org/browse/SI-8518).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1817] RDD.zip() should verify partition...

[witgo]

Github user witgo commented on the pull request:

https://github.com/apache/spark/pull/760#issuecomment-43396698
  
@kanzhang 
All told, we should fix the following code
 `slices += r.take(sliceSize).asInstanceOf[Seq[T]]`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1712]: TaskDescription instance is too ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/694#issuecomment-43396183
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12765889
  
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -303,14 +309,36 @@ object SparkSubmit {
   }
 
   /**
+   * Return whether the given primary resource represents a user jar.
+   */
+  private def isUserJar(primaryResource: String): Boolean = {
+!isShell(primaryResource) && !isPython(primaryResource)
+  }
+
+  /**
+   * Return whether the given primary resource represents a shell.
+   */
+  private def isShell(primaryResource: String): Boolean = {
+primaryResource == SPARK_SHELL || primaryResource == PYSPARK_SHELL
+  }
+
+  /**
+   * Return whether the given primary resource requires running python.
+   */
+  private[spark] def isPython(primaryResource: String): Boolean = {
+primaryResource.endsWith(".py") || primaryResource == PYSPARK_SHELL
+  }
+
+  /**
* Merge a sequence of comma-separated file lists, some of which may be 
null to indicate
* no files, into a single comma-separated string.
*/
   private[spark] def mergeFileLists(lists: String*): String = {
-val merged = lists.filter(_ != null)
-  .flatMap(_.split(","))
-  .mkString(",")
-if (merged == "") null else merged
+lists
+  .filter(_ != null)
+  .filter(_ != "")
+  .flatMap(_.split(","))
+  .mkString(",")
--- End diff --

This will now set files to the empty string instead of null even if there 
are no files present. To be conservative, would it make sense to change it 
back? I think there are null checks on this downstream when the option assinger 
is used. Could this change the behavior in some cases, e.g. by settting 
`spark.files` sys prop even when there are no files present.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Version bump of spark-ec2 scripts

[pwendell]

GitHub user pwendell opened a pull request:

https://github.com/apache/spark/pull/809

Version bump of spark-ec2 scripts

This will allow us to change things in spark-ec2 related to the 1.0 release.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pwendell/spark spark-ec2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/809.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #809


commit 59117fb71754154b371174a137c07a082341b72e
Author: Patrick Wendell 
Date:   2014-05-17T03:54:28Z

Version bump of spark-ec2 scripts




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Version bump of spark-ec2 scripts

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/809#issuecomment-43396879
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1712]: TaskDescription instance is too ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/694#issuecomment-43395354
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1837] NumericRange should be partitione...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/776#issuecomment-43395958
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/802#issuecomment-43395459
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/802#issuecomment-43396258
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15063/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43395353
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/802#issuecomment-43396257
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43396184
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12765879
  
--- Diff: bin/spark-shell ---
@@ -28,7 +28,7 @@ esac
 # Enter posix mode for bash
 set -o posix
 
-if [[ "$@" == *--help* ]]; then
+if [[ "$@" = *--help ]] || [[ "$@" = *--h ]]; then
--- End diff --

What is the second condition for? It doesn't seem to match `-h`. I noticed 
if I run this with `-h` then I seem to just get the help option from 
`spark-submit`.

```
patrick@patrick-t430s:~/Documents/spark$ ./bin/spark-shell -h
Usage: spark-submit [options]  [app options]
Options:
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1712]: TaskDescription instance is too ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/694#issuecomment-43396185
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15062/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1864 Look in spark conf instead of syste...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/808#issuecomment-43393840
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43395322
  
This is ready for merge from my side, provided that the tests pass (they 
should).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1729. Make Flume pull data from source, ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/807#issuecomment-43393108
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1837] NumericRange should be partitione...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/776#issuecomment-43395160
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/802#issuecomment-43395385
  
This is ready to merge (after #799)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43396186
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15061/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1864 Look in spark conf instead of syste...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/808


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1837] NumericRange should be partitione...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/776#issuecomment-43395959
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15060/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43393272
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43394310
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15058/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43395356
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1712]: TaskDescription instance is too ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/694#issuecomment-43395357
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1837] NumericRange should be partitione...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/776#issuecomment-43395161
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43391034
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1712]: TaskDescription instance is too ...

[witgo]

Github user witgo commented on a diff in the pull request:

https://github.com/apache/spark/pull/694#discussion_r12765545
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -140,8 +141,29 @@ class CoarseGrainedSchedulerBackend(scheduler: 
TaskSchedulerImpl, actorSystem: A
 // Launch tasks returned by a set of resource offers
 def launchTasks(tasks: Seq[Seq[TaskDescription]]) {
   for (task <- tasks.flatten) {
-freeCores(task.executorId) -= scheduler.CPUS_PER_TASK
-executorActor(task.executorId) ! LaunchTask(task)
+val ser = SparkEnv.get.closureSerializer.newInstance()
+val serializedTask = ser.serialize(task)
+if (serializedTask.limit >= akkaFrameSize - 1024) {
+  val taskSetId = scheduler.taskIdToTaskSetId(task.taskId)
+  scheduler.activeTaskSets.get(taskSetId).foreach { taskSet =>
+try {
+  var msg = "Serialized task %s:%d was %d bytes which " +
+"exceeds spark.akka.frameSize (%d bytes). " +
+"Consider using broadcast variables for large values."
+  msg = msg.format(task.taskId, task.index, 
serializedTask.limit, akkaFrameSize)
+  taskSet.abort(msg)
+} catch {
+  case e: Exception => logError("Exception in error callback", 
e)
+}
+  }
+  // scheduler.error(msg)
+  // TODO: Need to throw an exception?
+  // throw new SparkException(msg)
--- End diff --

Yes, it should be removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1864 Look in spark conf instead of syste...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/808#issuecomment-43394753
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Spark 1818

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/805#issuecomment-43390447
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15056/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/802#issuecomment-43395461
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1712]: TaskDescription instance is too ...

[mateiz]

Github user mateiz commented on a diff in the pull request:

https://github.com/apache/spark/pull/694#discussion_r12765502
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
 ---
@@ -140,8 +141,29 @@ class CoarseGrainedSchedulerBackend(scheduler: 
TaskSchedulerImpl, actorSystem: A
 // Launch tasks returned by a set of resource offers
 def launchTasks(tasks: Seq[Seq[TaskDescription]]) {
   for (task <- tasks.flatten) {
-freeCores(task.executorId) -= scheduler.CPUS_PER_TASK
-executorActor(task.executorId) ! LaunchTask(task)
+val ser = SparkEnv.get.closureSerializer.newInstance()
+val serializedTask = ser.serialize(task)
+if (serializedTask.limit >= akkaFrameSize - 1024) {
+  val taskSetId = scheduler.taskIdToTaskSetId(task.taskId)
+  scheduler.activeTaskSets.get(taskSetId).foreach { taskSet =>
+try {
+  var msg = "Serialized task %s:%d was %d bytes which " +
+"exceeds spark.akka.frameSize (%d bytes). " +
+"Consider using broadcast variables for large values."
+  msg = msg.format(task.taskId, task.index, 
serializedTask.limit, akkaFrameSize)
+  taskSet.abort(msg)
+} catch {
+  case e: Exception => logError("Exception in error callback", 
e)
+}
+  }
+  // scheduler.error(msg)
+  // TODO: Need to throw an exception?
+  // throw new SparkException(msg)
--- End diff --

Do we still need this code in here if we're aborting the TaskSet?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1837] NumericRange should be partitione...

[mateiz]

Github user mateiz commented on the pull request:

https://github.com/apache/spark/pull/776#issuecomment-43395120
  
Jenkins, this is ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1864 Look in spark conf instead of syste...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/808#issuecomment-43394754
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15059/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1864 Look in spark conf instead of syste...

[marmbrus]

GitHub user marmbrus opened a pull request:

https://github.com/apache/spark/pull/808

SPARK-1864 Look in spark conf instead of system properties when propagating 
configuration to executors.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/marmbrus/spark confClasspath

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/808.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #808


commit 4c31d57d37ac2766fcfa097f0700efbb9c2df0be
Author: Michael Armbrust 
Date:   2014-05-17T01:18:56Z

Look in spark conf instead of system properties when propagating 
configuration to executors.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1817] RDD.zip() should verify partition...

[witgo]

Github user witgo commented on the pull request:

https://github.com/apache/spark/pull/760#issuecomment-43394756
  
@kanzhang 

```
scala> sc.parallelize((1D to 2D).by(0.2),4).collect
res0: Array[Double] = Array(1.0, 1.2, 1.6, 1.8)
```
```
scala> sc.parallelize((1D to 2D).by(0.2),5).collect
res5: Array[Double] = Array(1.0, 1.2, 1.4, 1.5999, 
1.7998, 1.9998)
```
The first one should be wrong.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1864 Look in spark conf instead of syste...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/808#issuecomment-43393846
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1813. Add a utility to SparkConf that ma...

[mridulm]

Github user mridulm commented on the pull request:

https://github.com/apache/spark/pull/789#issuecomment-43388853
  
I have elaborated in the jira, but I will repeat it again for clarity:

This is adding an api for a specific case - it assumes single serialization
type (kryoserializable).
The reduction in boilerplate is not high, neither is it doing so for
something tricky/implementation detail.

Particularly given stress of simplifying our api this is going the opposite
direction without sufficient value to justify it.
 On 17-May-2014 4:34 am, "Sandy Ryza"  wrote:

> Do you mind elaborating on why it's a corner case? My understanding is
> that Kryo is the right choice over Java serialization in 99% of cases, and
> any time somebody wants to use their own objects with Kryo serialization,
> they need something like those 8 lines of code. I've had the opportunity 
to
> teach a bunch of people how to use Spark over the last few months and how
> to write those lines of code has come up almost every time.
>
> —
> Reply to this email directly or view it on 
GitHub
> .
>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[ash211]

Github user ash211 commented on a diff in the pull request:

https://github.com/apache/spark/pull/806#discussion_r12763329
  
--- Diff: docs/running-on-mesos.md ---
@@ -37,11 +36,25 @@ require any special patches of Mesos.
 If you already have a Mesos cluster running, you can skip this Mesos 
installation step.
 
 Otherwise, installing Mesos for Spark is no different than installing 
Mesos for use by other
-frameworks.  You can install Mesos using either prebuilt packages or by 
compiling from source.
+frameworks.  You can install Mesos either from source or using prebuilt 
packages.
--- End diff --

"... or using prebuilt packages from third parties."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43394309
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/806


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43393276
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Spark 1818

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/805#issuecomment-43390444
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43391035
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15057/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1729. Make Flume pull data from source, ...

[harishreedharan]

GitHub user harishreedharan opened a pull request:

https://github.com/apache/spark/pull/807

SPARK-1729. Make Flume pull data from source, rather than the current pu...

...sh model

Currently Spark uses Flume's internal Avro Protocol to ingest data from 
Flume. If the executor running the
receiver fails, it currently has to be restarted on the same node to be 
able to receive data.

This commit adds a new Sink which can be deployed to a Flume agent. This 
sink can be polled by a new
DStream that is also included in this commit. This model ensures that data 
can be pulled into Spark from
Flume even if the receiver is restarted on a new node. This also allows the 
receiver to receive data on
multiple threads for better performance.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harishreedharan/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/807.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #807


commit 6d6776a45f30e3594a15bda2582f99819c28a583
Author: Hari Shreedharan 
Date:   2014-05-09T06:16:56Z

SPARK-1729. Make Flume pull data from source, rather than the current push 
model

Currently Spark uses Flume's internal Avro Protocol to ingest data from 
Flume. If the executor running the
receiver fails, it currently has to be restarted on the same node to be 
able to receive data.

This commit adds a new Sink which can be deployed to a Flume agent. This 
sink can be polled by a new
DStream that is also included in this commit. This model ensures that data 
can be pulled into Spark from
Flume even if the receiver is restarted on a new node. This also allows the 
receiver to receive data on
multiple threads for better performance.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[mateiz]

Github user mateiz commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43394087
  
Changes look good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1846 Ignore logs directory in RAT checks

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/793#issuecomment-43242879
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1813. Add a utility to SparkConf that ma...

[ash211]

Github user ash211 commented on the pull request:

https://github.com/apache/spark/pull/789#issuecomment-43388939
  
Alternative to adding to `SparkContext`'s API, how about an API along the 
lines of:

```
Seq[Class[Any]] classes = ...
KryoRegistrator.enableKryoWithClasses(sc, classes)
```

The `enableKryoWithClasses` call would basically do the same thing as 
before but with a helper function.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Spark 1818

[mateiz]

Github user mateiz commented on the pull request:

https://github.com/apache/spark/pull/805#issuecomment-43388296
  
Ah, sorry, I was actually working on this in parallel: 
https://github.com/apache/spark/pull/806. Mind closing this PR and commenting 
on that one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[mateiz]

Github user mateiz commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43392397
  
Great, thanks for taking a look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12760990
  
--- Diff: python/pyspark/java_gateway.py ---
@@ -76,3 +76,30 @@ def run(self):
 java_import(gateway.jvm, "scala.Tuple2")
 
 return gateway
+
+def split_preserve_quotes(args):
--- End diff --

I just found out about a nice python library that does this 
(`shlex.split(string)`)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12727045
  
--- Diff: bin/pyspark ---
@@ -51,14 +57,20 @@ export 
PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.1-src.zip:$PYTHONPATH
 # Load the PySpark shell.py script when ./pyspark is used interactively:
 export OLD_PYTHONSTARTUP=$PYTHONSTARTUP
 export PYTHONSTARTUP=$FWDIR/python/pyspark/shell.py
+export PYSPARK_SUBMIT_ARGS="$@"
 
 if [ -n "$IPYTHON_OPTS" ]; then
   IPYTHON=1
 fi
 
-# Only use ipython if no command line arguments were provided [SPARK-1134]
-if [[ "$IPYTHON" = "1" && $# = 0 ]] ; then
-  exec ipython $IPYTHON_OPTS
+# If a python file is provided, directly run spark-submit
+if [[ "$1" =~ \.py$ ]]; then
+  exec $FWDIR/bin/spark-submit $PYSPARK_SUBMIT_ARGS
--- End diff --

This won't work with quoted arguments. The problem is that when you convert 
`$@` to a variable the type changes to a string from an array. Check out the 
way `ORIG_ARGS` is handled inside of `spark-submit`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[mateiz]

Github user mateiz commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43388350
  
I also changed "distribution" to "binary package" here to make it more 
consistent with the wording on our website.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43389137
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[mateiz]

Github user mateiz commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12760706
  
--- Diff: bin/pyspark ---
@@ -52,13 +58,35 @@ export 
PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.1-src.zip:$PYTHONPATH
 export OLD_PYTHONSTARTUP=$PYTHONSTARTUP
 export PYTHONSTARTUP=$FWDIR/python/pyspark/shell.py
 
+# If IPython options are specified, assume user wants to run IPython
 if [ -n "$IPYTHON_OPTS" ]; then
   IPYTHON=1
 fi
 
-# Only use ipython if no command line arguments were provided [SPARK-1134]
-if [[ "$IPYTHON" = "1" && $# = 0 ]] ; then
-  exec ipython $IPYTHON_OPTS
+# Build up arguments list manually to preserve quotes. We export Spark 
submit arguments as an
+# environment variable because shell.py must run as a PYTHONSTARTUP 
script, which does not take
+# in arguments. This is required mainly for IPython notebooks.
+
+PYSPARK_SUBMIT_ARGS=""
+whitespace="[[:space:]]"
+for i in "$@"; do
+  if [[ $i =~ $whitespace ]]; then
+i=\"$i\"
--- End diff --

Actually I tried it and it works


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: bugfix: overflow of graphx Edge compare functi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/769#issuecomment-43279657
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43390446
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15055/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43390445
  
Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43389131
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1818 Freshen Mesos documentation

[ash211]

Github user ash211 commented on a diff in the pull request:

https://github.com/apache/spark/pull/756#discussion_r12762858
  
--- Diff: docs/running-on-mesos.md ---
@@ -3,19 +3,123 @@ layout: global
 title: Running Spark on Mesos
 ---
 
-Spark can run on clusters managed by [Apache 
Mesos](http://mesos.apache.org/). Follow the steps below to install Mesos and 
Spark:
-
-1. Download and build Spark using the instructions [here](index.html). 
**Note:** Don't forget to consider what version of HDFS you might want to use!
-2. Download, build, install, and start Mesos {{site.MESOS_VERSION}} on 
your cluster. You can download the Mesos distribution from a 
[mirror](http://www.apache.org/dyn/closer.cgi/mesos/{{site.MESOS_VERSION}}/). 
See the Mesos [Getting Started](http://mesos.apache.org/gettingstarted) page 
for more information. **Note:** If you want to run Mesos without installing it 
into the default paths on your system (e.g., if you don't have administrative 
privileges to install it), you should also pass the `--prefix` option to 
`configure` to tell it where to install. For example, pass 
`--prefix=/home/user/mesos`. By default the prefix is `/usr/local`.
-3. Create a Spark "distribution" using `make-distribution.sh`.
-4. Rename the `dist` directory created from `make-distribution.sh` to 
`spark-{{site.SPARK_VERSION}}`.
-5. Create a `tar` archive: `tar czf spark-{{site.SPARK_VERSION}}.tar.gz 
spark-{{site.SPARK_VERSION}}`
-6. Upload this archive to HDFS or another place accessible from Mesos via 
`http://`, e.g., [Amazon Simple Storage Service](http://aws.amazon.com/s3): 
`hadoop fs -put spark-{{site.SPARK_VERSION}}.tar.gz 
/path/to/spark-{{site.SPARK_VERSION}}.tar.gz`
-7. Create a file called `spark-env.sh` in Spark's `conf` directory, by 
copying `conf/spark-env.sh.template`, and add the following lines to it:
-   * `export MESOS_NATIVE_LIBRARY=`. This path is 
usually `/lib/libmesos.so` (where the prefix is `/usr/local` by 
default, see above). Also, on Mac OS X, the library is called `libmesos.dylib` 
instead of `libmesos.so`.
-   * `export SPARK_EXECUTOR_URI=`.
-   * `export MASTER=mesos://HOST:PORT` where HOST:PORT is the host and 
port (default: 5050) of your Mesos master (or `zk://...` if using Mesos with 
ZooKeeper).
-8. To run a Spark application against the cluster, when you create your 
`SparkContext`, pass the string `mesos://HOST:PORT` as the master URL. In 
addition, you'll need to set the `spark.executor.uri` property. For example:
+# Why Mesos
+
+Spark can run on hardware clusters managed by [Apache 
Mesos](http://mesos.apache.org/).
+
+The advantages of deploying Spark with Mesos include:
+- dynamic partitioning between Spark and other
+  
[frameworks](https://mesos.apache.org/documentation/latest/mesos-frameworks/)
+- scalable partitioning between multiple instances of Spark
+
+# How it works
+
+In a standalone cluster deployment, the cluster manager in the below 
diagram is a Spark master
+instance.  When using Mesos, the Mesos master replaces the Spark master as 
the cluster manager.
+
+
+  
+
+
+Now when a driver creates a job and starts issuing tasks for scheduling, 
Mesos determines what
+machines handle what tasks.  Because it takes into account other 
frameworks when scheduling these
+many short-lived tasks, multiple frameworks can coexist on the same 
cluster without resorting to a
+static partitioning of resources.
+
+To get started, follow the steps below to install Mesos and deploy Spark 
jobs via Mesos.
+
+
+# Installing Mesos
+
+Spark {{site.SPARK_VERSION}} is designed for use with Mesos 
{{site.MESOS_VERSION}} and does not
+require any special patches of Mesos.
+
+If you already have a Mesos cluster running, you can skip this Mesos 
installation step.
+
+Otherwise, installing Mesos for Spark is no different than installing 
Mesos for use by other
+frameworks.  You can install Mesos using either prebuilt packages or by 
compiling from source.
+
+## Prebuilt packages
+
+The Apache Mesos project only publishes source package releases, no binary 
releases.  But other
+third party projects publish binary releases that may be helpful in 
setting Mesos up.
+
+One of those is Mesosphere.  To install Mesos using the binary releases 
provided by Mesosphere:
--- End diff --

https://github.com/apache/spark/pull/805


On Fri, May 16, 2014 at 3:41 PM, Andrew Ash  wrote:

> Got it.  I'm starting a cleanup PR that should be in within an hour.
>
>
> On Fri, May 16, 2014 at 3:17 PM, Matei Zaharia 
wrote:
>
>> In docs/running-on-mesos.md:
>>
>> > +# Installing Mesos
>> > +
>> > +Spark {{site.SPARK_VERSION}} is designed for use with Mesos 
{{site.MESOS_VERSION}} and does not
>> >

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43388331
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[ash211]

Github user ash211 commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43388649
  
I was using distribution because the script that creates it is called 
`make-distribution.sh`.  Maybe that should be renamed.  Should I file a jira to 
rename that to `make-binary-package.sh`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SQL] Fix tiny/small ints from HiveMetastore.

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/797#issuecomment-43270726
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: bugfix: overflow of graphx Edge compare functi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/769#issuecomment-43303541
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43389659
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15054/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1845] [SQL] Use AllScalaRegistrar for S...

[ueshin]

Github user ueshin closed the pull request at:

https://github.com/apache/spark/pull/790


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: improve performance of MemoryStore#tryToPut by...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/791#issuecomment-43200993
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43388326
  
 Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Spark 1818

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/805#issuecomment-43388333
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Spark 1818

[ash211]

Github user ash211 commented on the pull request:

https://github.com/apache/spark/pull/805#issuecomment-43388573
  
Will follow up there, thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[mateiz]

Github user mateiz commented on a diff in the pull request:

https://github.com/apache/spark/pull/802#discussion_r12723519
  
--- Diff: examples/src/main/python/logistic_regression.py ---
@@ -47,12 +47,12 @@ def readPointBatch(iterator):
 return [matrix]
 
 if __name__ == "__main__":
-if len(sys.argv) != 4:
-print >> sys.stderr, "Usage: logistic_regression   
"
+if len(sys.argv) != 3:
+print >> sys.stderr, "Usage: logistic_regression  
"
 exit(-1)
-sc = SparkContext(sys.argv[1], "PythonLR", 
pyFiles=[realpath(__file__)])
-points = sc.textFile(sys.argv[2]).mapPartitions(readPointBatch).cache()
-iterations = int(sys.argv[3])
+sc = SparkContext(appName="PythonLR", pyFiles=[realpath(__file__)])
--- End diff --

Same here, don't need pyFiles


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Tweaks to Mesos docs

[ash211]

Github user ash211 commented on the pull request:

https://github.com/apache/spark/pull/806#issuecomment-43389197
  
+1 LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1824] Remove from Python examp...

[andrewor14]

GitHub user andrewor14 opened a pull request:

https://github.com/apache/spark/pull/802

[SPARK-1824] Remove  from Python examples

A recent PR (#552) fixed this for all Scala / Java examples. We need to do 
it for python too.

Note that this blocks on #799, which makes `bin/pyspark` go through Spark 
submit. With only the changes in this PR, the only way to run these examples is 
through Spark submit. Once #799 goes in, you can use `bin/pyspark` to run them 
too. For example,

```
bin/pyspark examples/src/main/python/pi.py 100 --master 
local-cluster[4,1,512]
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewor14/spark python-examples

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/802.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #802


commit d32072c0bd60b1256a1c95bffcd1264bc80a4594
Author: Andrew Or 
Date:   2014-05-16T01:32:37Z

Remove  from examples + update usages




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1862: Support for MapR in the Maven buil...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/803#issuecomment-43299372
  
All automated tests passed.
Refer to this link for build results: 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15042/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43286480
  
Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SQL] Fix tiny/small ints from HiveMetastore.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/797


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/799#issuecomment-43367475
  
Great catch. Looks like `--die-on-broken-pipe` was in fact not propagated 
(see commit message for more details).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [Spark-1461] Deferred Expression Evaluation (s...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/446#issuecomment-43287344
  
Thanks @rxin.
It's done, can you re-test it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Spark 1818

[ash211]

GitHub user ash211 opened a pull request:

https://github.com/apache/spark/pull/805

Spark 1818



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ash211/spark SPARK-1818

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/805.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #805


commit 19e61c5aae2774ec7ffad348ee51eaf41e47b87c
Author: Andrew Ash 
Date:   2014-05-16T22:46:16Z

Put Apache Mesos section above Mesosphere prebuilt package section

commit bbe1f6f549e6d81299e61b25a5338d96b9742bd5
Author: Andrew Ash 
Date:   2014-05-16T22:54:33Z

Emphasize third party vs Apache more heavily




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[mateiz]

Github user mateiz commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12723642
  
--- Diff: bin/pyspark ---
@@ -25,6 +25,12 @@ export SPARK_HOME="$FWDIR"
 
 SCALA_VERSION=2.10
 
+if [[ "$@" == *--help* ]]; then
+  echo "Usage: ./bin/pyspark [python file] [options]"
--- End diff --

If we want to deprecate submitting a Python file with this, drop the 
`[python file]`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-1851. Upgrade Avro dependency to 1.7.6 s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/795#issuecomment-43258983
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Spark 1818

[ash211]

Github user ash211 closed the pull request at:

https://github.com/apache/spark/pull/805


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-1808] Route bin/pyspark through Spark s...

[mateiz]

Github user mateiz commented on a diff in the pull request:

https://github.com/apache/spark/pull/799#discussion_r12759418
  
--- Diff: bin/pyspark ---
@@ -52,13 +58,35 @@ export 
PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.1-src.zip:$PYTHONPATH
 export OLD_PYTHONSTARTUP=$PYTHONSTARTUP
 export PYTHONSTARTUP=$FWDIR/python/pyspark/shell.py
 
+# If IPython options are specified, assume user wants to run IPython
 if [ -n "$IPYTHON_OPTS" ]; then
   IPYTHON=1
 fi
 
-# Only use ipython if no command line arguments were provided [SPARK-1134]
-if [[ "$IPYTHON" = "1" && $# = 0 ]] ; then
-  exec ipython $IPYTHON_OPTS
+# Build up arguments list manually to preserve quotes. We export Spark 
submit arguments as an
+# environment variable because shell.py must run as a PYTHONSTARTUP 
script, which does not take
+# in arguments. This is required mainly for IPython notebooks.
+
+PYSPARK_SUBMIT_ARGS=""
+whitespace="[[:space:]]"
+for i in "$@"; do
+  if [[ $i =~ $whitespace ]]; then
+i=\"$i\"
--- End diff --

Does this work if the argument contains quote characters?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Optionally include Hive as a dependency of the...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/801#issuecomment-43285131
  
Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---