[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-13 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26401382 --- Diff: bin/pyspark --- @@ -95,26 +83,13 @@ export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH # Load the PySpark shell.py

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26361718 --- Diff: bin/pyspark --- @@ -95,26 +83,13 @@ export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH # Load the PySpark

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-12 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26320982 --- Diff: bin/pyspark --- @@ -95,26 +83,13 @@ export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH # Load the PySpark shell.py

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26358490 --- Diff: bin/pyspark --- @@ -95,26 +83,13 @@ export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH # Load the PySpark

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-11 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-78219718 Okay cool - LGTM I will pull this in. I just did some local sanity tests, built with Maven and ran (a few) Maven tests. We'll need to keep any eye on the Maven build

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3916 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-11 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26279970 --- Diff: bin/pyspark --- @@ -95,26 +83,13 @@ export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH # Load the PySpark

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-78182670 [Test build #28447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28447/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-78182678 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-10 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-78143279 Yeah this LGTM. There are a few other minor comments I left regarding visibility but they don't have to block this PR from going in. I haven't tested whether all the

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-10 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-78003405 @andrewor14 are you good with this one? I'd like to merge it soon! --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-10 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-78158391 FYI I ran our test suite with this change (covers standalone/client, yarn/both, spark-shell and pyspark, Linux only) and all looks ok. Also did some manual testing on

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-78171114 [Test build #28447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28447/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079948 --- Diff: bin/spark-sql --- @@ -43,15 +46,12 @@ function usage { echo echo CLI options: $FWDIR/bin/spark-class $CLASS --help 21 | grep

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079926 --- Diff: bin/spark-sql --- @@ -25,12 +25,15 @@ set -o posix # NOTE: This exact class name is matched downstream by SparkSubmit. # Any

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079987 --- Diff: bin/spark-submit --- @@ -17,58 +17,18 @@ # limitations under the License. # -# NOTE: Any changes in this file must be reflected

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079978 --- Diff: bin/spark-submit --- @@ -17,58 +17,18 @@ # limitations under the License. # -# NOTE: Any changes in this file must be reflected

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26080561 --- Diff: sbin/spark-daemon.sh --- @@ -121,45 +121,63 @@ if [ $SPARK_NICENESS = ]; then export SPARK_NICENESS=0 fi +run_command() {

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26080617 --- Diff: sbin/spark-daemon.sh --- @@ -121,45 +121,63 @@ if [ $SPARK_NICENESS = ]; then export SPARK_NICENESS=0 fi +run_command() {

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77929101 Hey Marcelo, I think this is really close to being ready to merge. I noticed some extra output (a '0') when running usage on `spark-shell`: ```

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26080422 --- Diff: sbin/spark-daemon.sh --- @@ -121,45 +121,63 @@ if [ $SPARK_NICENESS = ]; then export SPARK_NICENESS=0 fi +run_command() {

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26081687 --- Diff: bin/pyspark2.cmd --- @@ -17,59 +17,22 @@ rem See the License for the specific language governing permissions and rem limitations under the

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77954145 Sorry to spam the PR with comments about Bash word splitting, @vanzin. I think what you have Bash-wise looks good, but as a matter of consistency I just went through

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079243 --- Diff: bin/spark-class --- @@ -110,83 +39,48 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 21 | grep 'version' | sed

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26078786 --- Diff: bin/pyspark2.cmd --- @@ -17,59 +17,22 @@ rem See the License for the specific language governing permissions and rem limitations under the

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26078620 --- Diff: bin/pyspark --- @@ -18,36 +18,24 @@ # # Figure out where Spark is installed -FWDIR=$(cd `dirname $0`/..; pwd) +export

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079792 --- Diff: bin/spark-class --- @@ -110,83 +39,48 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 21 | grep 'version' | sed

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079694 --- Diff: bin/spark-class --- @@ -110,83 +39,48 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 21 | grep 'version' | sed 's/.*

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079633 --- Diff: bin/spark-class --- @@ -110,83 +39,48 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 21 | grep 'version' | sed

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079837 --- Diff: bin/spark-shell --- @@ -28,25 +28,24 @@ esac # Enter posix mode for bash set -o posix -## Global script variables

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079752 --- Diff: bin/spark-class --- @@ -110,83 +39,48 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 21 | grep 'version' | sed 's/.*

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26080472 --- Diff: sbin/spark-daemon.sh --- @@ -121,45 +121,63 @@ if [ $SPARK_NICENESS = ]; then export SPARK_NICENESS=0 fi +run_command() {

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26081000 --- Diff: bin/spark-class --- @@ -110,83 +39,48 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 21 | grep 'version' | sed

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26080991 --- Diff: bin/spark-class --- @@ -110,83 +39,48 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 21 | grep 'version' | sed

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079852 --- Diff: bin/spark-shell --- @@ -28,25 +28,24 @@ esac # Enter posix mode for bash set -o posix -## Global script variables

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26081238 --- Diff: sbin/start-thriftserver.sh --- @@ -52,4 +52,4 @@ fi export SUBMIT_USAGE_FUNCTION=usage -exec $FWDIR/sbin/spark-daemon.sh

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26078579 --- Diff: bin/pyspark --- @@ -18,36 +18,24 @@ # # Figure out where Spark is installed -FWDIR=$(cd `dirname $0`/..; pwd) +export

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26079464 --- Diff: bin/pyspark2.cmd --- @@ -17,59 +17,22 @@ rem See the License for the specific language governing permissions and rem limitations under the

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26081219 --- Diff: sbin/spark-daemon.sh --- @@ -121,45 +121,63 @@ if [ $SPARK_NICENESS = ]; then export SPARK_NICENESS=0 fi +run_command() {

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77936105 [Test build #28401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28401/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77935980 [Test build #28401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28401/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77937027 [Test build #28403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28403/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77937171 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77937990 [Test build #28404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28404/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26065985 --- Diff: launcher/src/main/java/org/apache/spark/launcher/Main.java --- @@ -0,0 +1,173 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77936109 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26067056 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -0,0 +1,327 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r26067102 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -0,0 +1,327 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77937167 [Test build #28403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28403/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77970428 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77970418 [Test build #28407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28407/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77958210 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77957236 [Test build #28407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28407/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77958202 [Test build #28404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28404/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-04 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-77180148 FYI: There is [a question on SO](http://stackoverflow.com/q/28841940/877069), I believe, about the type of functionality being added in this PR. --- If your project

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76849555 [Test build #28200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28200/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76875689 [Test build #28205 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28205/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76875697 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76864166 [Test build #28205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28205/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76862929 [Test build #28200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28200/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76862941 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76293224 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76188802 Sorry I haven't had time to look at this in detail, one thing I was looking for though was some documentation for the high level api. I realize java doc will document

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76230854 I'm ok with it being separate, although in general I think its bad practice. I've seen some features go in without documentation which I think as a user is a major

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76227779 @tgravescs that can definitely be added, but I'd rather do it separately at this point. Also, regarding javadoc, I'm not sure how it's generally published for Spark: the

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76275450 [Test build #28016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28016/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76276321 [Test build #28017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28017/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76278354 [Test build #28018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28018/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76296105 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76296090 [Test build #28017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28017/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76296505 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76296499 [Test build #28018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28018/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76293215 [Test build #28016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28016/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76054123 [Test build #27958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27958/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25394351 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitOptionParser.java --- @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25396504 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -0,0 +1,335 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400181 --- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java --- @@ -0,0 +1,359 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400174 --- Diff: launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java --- @@ -0,0 +1,228 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400811 --- Diff: bin/pyspark --- @@ -95,26 +80,13 @@ export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH # Load the PySpark

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76114272 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76114251 @vanzin @pwendell As of the latest changes I think this PR is almost ready to be merged. An obvious blocker is that the `SparkLauncher` needs to also handle Windows,

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76115100 [Test build #27971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27971/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76115110 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76107021 [Test build #27972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27972/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25399842 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -0,0 +1,335 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400927 --- Diff: core/src/main/scala/org/apache/spark/launcher/WorkerCommandBuilder.scala --- @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25401107 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java --- @@ -0,0 +1,270 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76113760 [Test build #27972 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27972/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76113766 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76114901 `SparkLauncher` works on Windows. Did you see anything that indicates otherwise? I even managed to get all unit tests under launcher/ to pass on Windows (although I

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25399597 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -0,0 +1,335 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400090 --- Diff: launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java --- @@ -0,0 +1,228 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76114681 [Test build #27977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27977/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400728 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java --- @@ -0,0 +1,270 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400989 --- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java --- @@ -0,0 +1,362 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25400987 --- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java --- @@ -0,0 +1,362 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76076202 [Test build #27958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27958/consoleFull) for PR 3916 at commit

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76076214 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25388445 --- Diff: launcher/src/main/java/org/apache/spark/launcher/Main.java --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-25 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3916#discussion_r25394241 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -0,0 +1,335 @@ +/* + * Licensed to the Apache

  1   2   3   4   5   >