[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/6870#issuecomment-136954542
  
Thanks @andrewor14 , one of my colleague will continue to work on this :).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10261][Documentation, ML] Fixed @Since ...

[tijoparacka]

Github user tijoparacka commented on a diff in the pull request:

https://github.com/apache/spark/pull/8554#discussion_r38502967
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala
 ---
@@ -31,15 +31,18 @@ import org.apache.spark.sql.types.DoubleType
  * Evaluator for binary classification, which expects two input columns: 
rawPrediction and label.
  */
 @Experimental
+@Since("1.2.0")
 class BinaryClassificationEvaluator(override val uid: String)
--- End diff --

 i tried with following comand 
spark@master:~/sparktijo/spark$ git show 
v1.4.0:mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala
 | grep  "def this() = this"
this shows the result as matching  but with 1.2.0 it is not giving any 
result for the above command.  Could you please  confirm.



  def this() = this(Identifiable.randomUID("binEval"))



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10310][SQL]Using \t as the field delime...

[jameszhouyi]

Github user jameszhouyi commented on the pull request:

https://github.com/apache/spark/pull/8476#issuecomment-13695
  
This is real-world case using Spark SQL and hopefully it can be 
fixed/merged in Spark 1.5.0.Thanks in advance !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9718][ML] linear regression training su...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8564#issuecomment-136954162
  
  [Test build #41915 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41915/consoleFull)
 for   PR 8564 at commit 
[`43420b7`](https://github.com/apache/spark/commit/43420b7a14ff88f7d6aa3f288093537a8e52502f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9718][ML] linear regression training su...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8564#issuecomment-136951779
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9718][ML] linear regression training su...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8564#issuecomment-136951696
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9718][ML] linear regression training su...

[holdenk]

GitHub user holdenk opened a pull request:

https://github.com/apache/spark/pull/8564

[SPARK-9718][ML] linear regression training summary all columns

LinearRegression training summary: The transformed dataset should hold all 
columns, not just selected ones like prediction and label. There is no real 
need to remove some, and the user may find them useful.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/spark 
SPARK-9718-LinearRegressionTrainingSummary-all-columns

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8564.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8564


commit 6afde57ebf3d86ae3a0bb07cdb42875a7bbbd0ff
Author: Holden Karau 
Date:   2015-09-02T06:09:26Z

switch to withColumn

commit 43420b7a14ff88f7d6aa3f288093537a8e52502f
Author: Holden Karau 
Date:   2015-09-02T06:35:43Z

Fix it to work and add a test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4122][STREAMING] Add a library that can...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/2994#issuecomment-136950364
  
Sure, if there is interest in this. I have a version of this on the 
cloudera github (which I pushed there, since I didn't see much interest in 
having this in spark itself). I will port it over.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[nilmeier]

Github user nilmeier commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136949480
  
Thanks Xiangrui!  Cheers, J

Sent from my iPhone

> On Sep 1, 2015, at 11:22 PM, Xiangrui Meng  
wrote:
> 
> ok to test
> 
> —
> Reply to this email directly or view it on GitHub.
> 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136949391
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136949392
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41911/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136949303
  
  [Test build #41911 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41911/console)
 for   PR 7884 at commit 
[`e94bb95`](https://github.com/apache/spark/commit/e94bb9559b98f1ae9e58cf88eddc0415a509687e).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  final val probabilityCol: Param[String] = new Param[String](this, 
"probabilityCol", "Column name for predicted class conditional probabilities. 
Note: Not all models output well-calibrated probability estimates! These 
probabilities should be treated as confidences, not precise probabilities")`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136948993
  
  [Test build #41914 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41914/consoleFull)
 for   PR 8563 at commit 
[`2f62cca`](https://github.com/apache/spark/commit/2f62cca83ca8ba4a21db5bf78a776743b93a6021).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7336][HistoryServer] Fix bug that appli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5886#issuecomment-136948621
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41909/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7336][HistoryServer] Fix bug that appli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5886#issuecomment-136948619
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136948527
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136948220
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136948581
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7336][HistoryServer] Fix bug that appli...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5886#issuecomment-136948316
  
  [Test build #41909 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41909/console)
 for   PR 5886 at commit 
[`a4a876a`](https://github.com/apache/spark/commit/a4a876a2ada3a949bf7812b6c3dc83f99dc7dc27).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136948170
  
add to whitelist


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4962] [CORE] Put TaskScheduler.start ba...

[YanTangZhai]

Github user YanTangZhai closed the pull request at:

https://github.com/apache/spark/pull/3810


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136946511
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41912/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136946510
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136946464
  
  [Test build #41912 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41912/console)
 for   PR 4027 at commit 
[`7811cd0`](https://github.com/apache/spark/commit/7811cd041f8580dcb68f4f4db46532b112d79742).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[nilmeier]

Github user nilmeier commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136944803
  
I have been corresponding with Xiangrui Meng on this.  (I'm with IBM
nilme...@us.ibm.com).  I also sent him an email just now.  Cheers, Jerome

On Tue, Sep 1, 2015 at 10:38 PM, UCB AMPLab 
wrote:

> Can one of the admins verify this patch?
>
> —
> Reply to this email directly or view it on GitHub
> .
>



-- 
Jerome Nilmeier, PhD
Cell:  510-325-8695
Home:   925-292-5321



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...

[sarutak]

Github user sarutak closed the pull request at:

https://github.com/apache/spark/pull/5437


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/5437#issuecomment-136944437
  
Thanks for the notification. Unfortunately, now I don't have time to 
improve this patch. So I'll close for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7927#issuecomment-136942619
  
  [Test build #41907 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41907/console)
 for   PR 7927 at commit 
[`6c6d53d`](https://github.com/apache/spark/commit/6c6d53d61d293e2c14ed3f776b8fb20397a6b332).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class BlockFetchException(messages: String, throwable: Throwable)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7927#issuecomment-136942750
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7927#issuecomment-136942752
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41907/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10288][Yarn] Add REST client for Spark ...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/8504#issuecomment-136942179
  
Hi @andrewor14 , thanks a lot for your comment. To some extent, this PR 
provides an additional choice for user to submit application on Yarn. For users 
they want to build a gateway to submit applications, this rest client will be 
easy for them to do it.

I admitted this PR will increase the complexity of current yarn/client 
code, not sure what is your opinion and suggestion on it?

Thanks a lot.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-136941559
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[nilmeier]

GitHub user nilmeier opened a pull request:

https://github.com/apache/spark/pull/8563

[SPARK-8514] LU factorization on BlockMatrix

@mengxr and/or @shivaram may be reviewing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nilmeier/spark SPARK-8514_LU_factorization

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8563.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8563


commit 9214a1c3891a87c96ce77f17c3197f6d5984c210
Author: jerome nilmeier 
Date:   2015-08-27T20:52:56Z

SPARK-8514 First Commit

commit ed3bcd45e4edd55b8335f1cfcbaca2d67dbc1d52
Author: jerome nilmeier 
Date:   2015-08-31T00:41:51Z

Ffixed Permutation Matrix isse.

commit 6ba061431cd6402748efdf18b6cd27c299a334a0
Author: jerome nilmeier 
Date:   2015-09-01T23:41:44Z

Added documentation.  Also made some methods private and removed unused 
methods in BlockMatrix.  Added one more unit test

commit 2953f1932a10db97698a368e902b7b744ed5f0a7
Author: jerome nilmeier 
Date:   2015-09-01T23:42:57Z

removed extraneous doc

commit 49a3cdbcc2ba5bf7d6b77a850e063d5774232e88
Author: jerome nilmeier 
Date:   2015-09-02T02:54:20Z

updating from apache

commit 2f62cca83ca8ba4a21db5bf78a776743b93a6021
Author: jerome nilmeier 
Date:   2015-09-02T05:08:44Z

wRan ./dev/run-tests and fixed up style.  Also modified docs to have 
example near the top of the page.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/5385#issuecomment-136940195
  
OK, thanks a lot. Will port if still make sense.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...

[jerryshao]

Github user jerryshao closed the pull request at:

https://github.com/apache/spark/pull/5385


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/8349#discussion_r38499639
  
--- Diff: docs/running-on-mesos.md ---
@@ -157,6 +157,9 @@ From the client, you can submit a job to Mesos cluster 
by running `spark-submit`
 to the url of the MesosClusterDispatcher (e.g: mesos://dispatcher:7077). 
You can view driver statuses on the
 Spark cluster Web UI.
 
+Note that jars or python files that are passed to spark-submit should be 
uris that is reachable
--- End diff --

typo: that *are* reachable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2926][Shuffle]Add MR style sort-merge s...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/3438#issuecomment-136939951
  
OK, thanks a lot.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2926][Shuffle]Add MR style sort-merge s...

[jerryshao]

Github user jerryshao closed the pull request at:

https://github.com/apache/spark/pull/3438


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/4027#discussion_r38499472
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
 ---
@@ -357,35 +379,39 @@ private[spark] class CoarseMesosSchedulerBackend(
* slave IDs that we might have asked to be killed. It also notifies the 
driver
* that an executor was removed.
*/
-  private def executorTerminated(d: SchedulerDriver, slaveId: String, 
reason: String): Unit = {
+  private def executorTerminated(
+  d: SchedulerDriver,
+  executorId: String,
+  slaveId: String,
+  reason: String): Unit = {
 stateLock.synchronized {
-  if (slaveIdsWithExecutors.contains(slaveId)) {
-val slaveIdToTaskId = taskIdToSlaveId.inverse()
-if (slaveIdToTaskId.containsKey(slaveId)) {
-  val taskId: Int = slaveIdToTaskId.get(slaveId)
-  taskIdToSlaveId.remove(taskId)
-  removeExecutor(sparkExecutorId(slaveId, taskId.toString), reason)
+  if (slaveIdsWithExecutors.contains(slaveId) && 
taskIdToSlaveId.contains(executorId)) {
+taskIdToSlaveId.remove(executorId)
+removeExecutor(executorId, reason)
+val newCount = slaveIdsWithExecutors(slaveId) - 1
+if (newCount == 0) {
+  slaveIdsWithExecutors.remove(slaveId)
+} else {
+  slaveIdsWithExecutors(slaveId) = newCount
 }
-// TODO: This assumes one Spark executor per Mesos slave,
-// which may no longer be true after SPARK-5095
-pendingRemovedSlaveIds -= slaveId
-slaveIdsWithExecutors -= slaveId
   }
 }
   }
 
-  private def sparkExecutorId(slaveId: String, taskId: String): String = {
-s"$slaveId/$taskId"
-  }
-
   override def slaveLost(d: SchedulerDriver, slaveId: SlaveID): Unit = {
 logInfo(s"Mesos slave lost: ${slaveId.getValue}")
-executorTerminated(d, slaveId.getValue, "Mesos slave lost: " + 
slaveId.getValue)
+// Terminate all executors in the slave
+stateLock.synchronized {
+  val lostExecutors = 
taskIdToSlaveId.filter(_._2.equals(slaveId.getValue)).map(_._1)
+  lostExecutors.foreach { taskId =>
+executorTerminated(d, taskId, slaveId.getValue, "Mesos slave lost: 
" + slaveId.getValue)
+  }
+}
   }
 
   override def executorLost(d: SchedulerDriver, e: ExecutorID, s: SlaveID, 
status: Int): Unit = {
-logInfo("Executor lost: %s, marking slave %s as 
lost".format(e.getValue, s.getValue))
-slaveLost(d, s)
+logInfo("Executor lost: %s".format(e.getValue))
--- End diff --

Nitpicking: you could use string interpolation: `s"Executor lost: 
${e.getValue}"`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136939354
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41908/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136939353
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6691][Streaming][WIP] Add a dynamic Rat...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/5385#issuecomment-136939299
  
I'm afraid not. The corresponding Jira ticket has been closed as a 
duplicated by @tdas on July 27, so we should close this PR as well. @jerryshao 
do you think there's something in your approach that would make sense to be 
ported to the existing backpressure implementation?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136939312
  
  [Test build #41908 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41908/console)
 for   PR 4027 at commit 
[`cc8d65b`](https://github.com/apache/spark/commit/cc8d65b3294910719d715a9a5e049f1f293b7f96).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `public class JavaTrainValidationSplitExample `
  * `class DCT(JavaTransformer, HasInputCol, HasOutputCol):`
  * `class SQLTransformer(JavaTransformer):`
  * `class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):`
  * `case class LimitNode(limit: Int, child: LocalNode) extends 
UnaryLocalNode `
  * `case class UnionNode(children: Seq[LocalNode]) extends LocalNode `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8101#issuecomment-136937602
  
  [Test build #41913 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41913/consoleFull)
 for   PR 8101 at commit 
[`fbc471b`](https://github.com/apache/spark/commit/fbc471bd8de9d7742b4329f79610ab9dd8fe123c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8101#issuecomment-136936960
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8101#issuecomment-136936968
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2750][WEB UI]Add Https support for Web ...

[jacek-lewandowski]

Github user jacek-lewandowski commented on the pull request:

https://github.com/apache/spark/pull/5664#issuecomment-136934630
  
@vanzin @andrewor14 I may take a look at this and finish or create a 
separate pr with my approach if you like.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10034][SQL] add regression test for Sor...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8231#issuecomment-136934030
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10034][SQL] add regression test for Sor...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8231#issuecomment-136934031
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41906/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10034][SQL] add regression test for Sor...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8231#issuecomment-136933970
  
  [Test build #41906 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41906/console)
 for   PR 8231 at commit 
[`845b161`](https://github.com/apache/spark/commit/845b1614ca7a38db4cc0e91c41d8d7b23556335c).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/7461#discussion_r38497342
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag](
   var partiallyAggregated = mapPartitions(it => 
Iterator(aggregatePartition(it)))
   var numPartitions = partiallyAggregated.partitions.length
   val scale = math.max(math.ceil(math.pow(numPartitions, 1.0 / 
depth)).toInt, 2)
+
+  // Do one level of aggregation based on executorId before starting 
the tree
+  // NOTE: exclude the driver from list of executors
+  val numExecutors = math.max(context.getExecutorStorageStatus.length 
- 1, 1)
+  partiallyAggregated = partiallyAggregated.mapPartitionsWithIndex { 
case (idx, iter) =>
+def isAllDigits(x: String) = x forall Character.isDigit
+val execId = SparkEnv.get.executorId
+if (isAllDigits(execId)) {
+  iter.map((execId.toInt, _))
+} else {
+  iter.map((execId.hashCode, _))
--- End diff --

Ah I can actually reproduce it -- so the problem is that the keyspace get 
un-balanced when we use hashCode on Strings. For example if you start with 100 
exec ids and want to get 50 partitions out, some partitions will get 3 ids 
while some will get 1
```
(0 until 100).map(_.toString).map(_.hashCode % 
50).groupBy(identity).map(_._2.size)
```
while with integers you get a uniform distribution
```
(0 until 100).map(_.hashCode % 50).groupBy(identity).map(_._2.size)
```

So you could end up with unbalanced trees which is not desirable. 

@andrewor14 Can we guarantee that execId will be an integer ? Is there any 
other integer we can get a handle on otherwise ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/7461#discussion_r38497048
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag](
   var partiallyAggregated = mapPartitions(it => 
Iterator(aggregatePartition(it)))
   var numPartitions = partiallyAggregated.partitions.length
   val scale = math.max(math.ceil(math.pow(numPartitions, 1.0 / 
depth)).toInt, 2)
+
+  // Do one level of aggregation based on executorId before starting 
the tree
+  // NOTE: exclude the driver from list of executors
+  val numExecutors = math.max(context.getExecutorStorageStatus.length 
- 1, 1)
+  partiallyAggregated = partiallyAggregated.mapPartitionsWithIndex { 
case (idx, iter) =>
+def isAllDigits(x: String) = x forall Character.isDigit
+val execId = SparkEnv.get.executorId
+if (isAllDigits(execId)) {
+  iter.map((execId.toInt, _))
+} else {
+  iter.map((execId.hashCode, _))
--- End diff --

Hmm it looks like the reason I added this in an earlier version was to 
avoid some hashCollisions[1]. Not sure what was exactly the problem as I can't 
reproduce any collisions with all digit strings -- so going with `hashCode` for 
now seems fine.

[1] 
https://github.com/amplab/ml-matrix/commit/5d7e00e136f8bbc2233adca6d495507d446815bf


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8101#issuecomment-136928983
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41903/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8101#issuecomment-136928977
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8101#issuecomment-136928856
  
  [Test build #41903 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41903/console)
 for   PR 8101 at commit 
[`de10e94`](https://github.com/apache/spark/commit/de10e94c0b22194cdb648d279945db8f583bcb69).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9990][SQL]Create local hash join operat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8535#issuecomment-136928705
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41902/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9990][SQL]Create local hash join operat...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8535#issuecomment-136928703
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136928679
  
  [Test build #41911 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41911/consoleFull)
 for   PR 7884 at commit 
[`e94bb95`](https://github.com/apache/spark/commit/e94bb9559b98f1ae9e58cf88eddc0415a509687e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9990][SQL]Create local hash join operat...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8535#issuecomment-136928604
  
  [Test build #41902 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41902/console)
 for   PR 8535 at commit 
[`aa928fd`](https://github.com/apache/spark/commit/aa928fd24d3fc0a316c438562998fe6b494bb20c).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class ConvertToSafeNode(conf: SQLConf, child: LocalNode) extends 
UnaryLocalNode(conf) `
  * `case class ConvertToUnsafeNode(conf: SQLConf, child: LocalNode) 
extends UnaryLocalNode(conf) `
  * `case class FilterNode(conf: SQLConf, condition: Expression, child: 
LocalNode)`
  * `case class HashJoinNode (`
  * `case class LimitNode(conf: SQLConf, limit: Int, child: LocalNode) 
extends UnaryLocalNode(conf) `
  * `abstract class LocalNode(conf: SQLConf) extends TreeNode[LocalNode] 
with Logging `
  * `abstract class LeafLocalNode(conf: SQLConf) extends LocalNode(conf) `
  * `abstract class UnaryLocalNode(conf: SQLConf) extends LocalNode(conf) `
  * `abstract class BinaryLocalNode(conf: SQLConf) extends LocalNode(conf) `
  * `case class ProjectNode(conf: SQLConf, projectList: 
Seq[NamedExpression], child: LocalNode)`
  * `case class SeqScanNode(conf: SQLConf, output: Seq[Attribute], data: 
Seq[InternalRow])`
  * `case class UnionNode(conf: SQLConf, children: Seq[LocalNode]) extends 
LocalNode(conf) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[tnachen]

Github user tnachen commented on a diff in the pull request:

https://github.com/apache/spark/pull/4027#discussion_r38496861
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
 ---
@@ -357,30 +380,34 @@ private[spark] class CoarseMesosSchedulerBackend(
* slave IDs that we might have asked to be killed. It also notifies the 
driver
* that an executor was removed.
*/
-  private def executorTerminated(d: SchedulerDriver, slaveId: String, 
reason: String): Unit = {
+  private def executorTerminated(
+  d: SchedulerDriver,
+  executorId: String,
+  slaveId: String,
+  reason: String): Unit = {
 stateLock.synchronized {
-  if (slaveIdsWithExecutors.contains(slaveId)) {
-val slaveIdToTaskId = taskIdToSlaveId.inverse()
-if (slaveIdToTaskId.containsKey(slaveId)) {
-  val taskId: Int = slaveIdToTaskId.get(slaveId)
-  taskIdToSlaveId.remove(taskId)
-  removeExecutor(sparkExecutorId(slaveId, taskId.toString), reason)
+  if (slaveIdsWithExecutors.contains(slaveId) && 
taskIdToSlaveId.contains(executorId)) {
+taskIdToSlaveId.remove(executorId)
+removeExecutor(executorId, reason)
+val newCount = slaveIdsWithExecutors(slaveId) - 1
+if (newCount == 0) {
+  slaveIdsWithExecutors.remove(slaveId)
+} else {
+  slaveIdsWithExecutors(slaveId) = newCount
 }
-// TODO: This assumes one Spark executor per Mesos slave,
-// which may no longer be true after SPARK-5095
-pendingRemovedSlaveIds -= slaveId
-slaveIdsWithExecutors -= slaveId
   }
 }
   }
 
-  private def sparkExecutorId(slaveId: String, taskId: String): String = {
-s"$slaveId/$taskId"
-  }
-
   override def slaveLost(d: SchedulerDriver, slaveId: SlaveID): Unit = {
 logInfo(s"Mesos slave lost: ${slaveId.getValue}")
-executorTerminated(d, slaveId.getValue, "Mesos slave lost: " + 
slaveId.getValue)
+// Terminate all executors in the slave
+stateLock.synchronized {
+  val lostExecutors = 
taskIdToSlaveId.filter(_._2.equals(slaveId.getValue)).map(_._1)
+  lostExecutors.foreach { taskId =>
+executorTerminated(d, taskId, slaveId.getValue, "Mesos slave lost: 
" + slaveId.getValue)
+  }
+}
   }
 
   override def executorLost(d: SchedulerDriver, e: ExecutorID, s: SlaveID, 
status: Int): Unit = {
--- End diff --

You're right, I've updated it now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136928560
  
  [Test build #41912 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41912/consoleFull)
 for   PR 4027 at commit 
[`7811cd0`](https://github.com/apache/spark/commit/7811cd041f8580dcb68f4f4db46532b112d79742).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136928429
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136928439
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136928425
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136928435
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8048] Partitionning of an RDD with 0 pa...

[kmadhugit]

Github user kmadhugit commented on the pull request:

https://github.com/apache/spark/pull/7020#issuecomment-136928343
  
@andrewor14, I can modify the code as per your suggestion to improve the 
readability and rebase it to master.

What about the concern raised by @JoshRosen ?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

[kmadhugit]

Github user kmadhugit commented on the pull request:

https://github.com/apache/spark/pull/7461#issuecomment-136928037
  
I've modified the title.

I'm facing a problem during unit test. When I call 
context.getExecutorStorageStatus.length it returns 1 instead of numExecutors + 
1(for driver). This happens because there is no prior job submitted to the 
executor by DAG, so the only registered block manager is the driver. We may 
need to find an alternative way to know number of executors(with an assumption 
that all executors will store some blocks of the RDD).  Without that change 
this fix won't work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136927334
  
  [Test build #41910 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41910/console)
 for   PR 7884 at commit 
[`a4c3e54`](https://github.com/apache/spark/commit/a4c3e543b950b2bd3c2001e97ef5546ea6c20308).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  final val probabilityCol: Param[String] = new Param[String](this, 
"probabilityCol", "Column name for predicted class conditional probabilities. 
Note: Not all models output well-calibrated probability estimates! These 
probabilities should be treated as confidences, not precise probabilities")`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136927349
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136927353
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41910/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

[kmadhugit]

Github user kmadhugit commented on a diff in the pull request:

https://github.com/apache/spark/pull/7461#discussion_r38496374
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag](
   var partiallyAggregated = mapPartitions(it => 
Iterator(aggregatePartition(it)))
   var numPartitions = partiallyAggregated.partitions.length
   val scale = math.max(math.ceil(math.pow(numPartitions, 1.0 / 
depth)).toInt, 2)
+
+  // Do one level of aggregation based on executorId before starting 
the tree
+  // NOTE: exclude the driver from list of executors
+  val numExecutors = math.max(context.getExecutorStorageStatus.length 
- 1, 1)
+  partiallyAggregated = partiallyAggregated.mapPartitionsWithIndex { 
case (idx, iter) =>
+def isAllDigits(x: String) = x forall Character.isDigit
+val execId = SparkEnv.get.executorId
+if (isAllDigits(execId)) {
+  iter.map((execId.toInt, _))
+} else {
+  iter.map((execId.hashCode, _))
--- End diff --

will do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8137][core] Improve treeAggregate to co...

[kmadhugit]

Github user kmadhugit commented on a diff in the pull request:

https://github.com/apache/spark/pull/7461#discussion_r38496331
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1080,6 +1080,21 @@ abstract class RDD[T: ClassTag](
   var partiallyAggregated = mapPartitions(it => 
Iterator(aggregatePartition(it)))
   var numPartitions = partiallyAggregated.partitions.length
   val scale = math.max(math.ceil(math.pow(numPartitions, 1.0 / 
depth)).toInt, 2)
+
+  // Do one level of aggregation based on executorId before starting 
the tree
+  // NOTE: exclude the driver from list of executors
+  val numExecutors = math.max(context.getExecutorStorageStatus.length 
- 1, 1)
--- End diff --

In that case numExecutors will become 1 and it will be aggregated in single 
pass similar to rdd.aggregate().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7336][HistoryServer] Fix bug that appli...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5886#issuecomment-136926610
  
  [Test build #41909 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41909/consoleFull)
 for   PR 5886 at commit 
[`a4a876a`](https://github.com/apache/spark/commit/a4a876a2ada3a949bf7812b6c3dc83f99dc7dc27).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136926599
  
  [Test build #41910 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41910/consoleFull)
 for   PR 7884 at commit 
[`a4c3e54`](https://github.com/apache/spark/commit/a4c3e543b950b2bd3c2001e97ef5546ea6c20308).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136926431
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7685][ML] Apply weights to different sa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7884#issuecomment-136926444
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4980] [MLlib] Add decay factors to stre...

[rotationsymmetry]

Github user rotationsymmetry commented on the pull request:

https://github.com/apache/spark/pull/8022#issuecomment-136924899
  
@feynmanliang I have make another push to the PR:

Refactor StreamingDecay
Use case object for TimeUnit
Clean up ScalaDoc

Thank again for your review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7336][HistoryServer] Fix bug that appli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5886#issuecomment-136924433
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7336][HistoryServer] Fix bug that appli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5886#issuecomment-136924395
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2827][GraphX] Add collectDegreeDist to ...

[ankurdave]

Github user ankurdave commented on the pull request:

https://github.com/apache/spark/pull/4399#issuecomment-136922400
  
LGTM aside from minor comments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136921658
  
  [Test build #41908 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41908/consoleFull)
 for   PR 4027 at commit 
[`cc8d65b`](https://github.com/apache/spark/commit/cc8d65b3294910719d715a9a5e049f1f293b7f96).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136920924
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP] [skip ci] Fuzz testing in Spark SQL

[JoshRosen]

Github user JoshRosen closed the pull request at:

https://github.com/apache/spark/pull/7625


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-136920879
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7927#issuecomment-136920246
  
  [Test build #41907 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41907/consoleFull)
 for   PR 7927 at commit 
[`6c6d53d`](https://github.com/apache/spark/commit/6c6d53d61d293e2c14ed3f776b8fb20397a6b332).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2827][GraphX] Add collectDegreeDist to ...

[ankurdave]

Github user ankurdave commented on a diff in the pull request:

https://github.com/apache/spark/pull/4399#discussion_r38495441
  
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/GraphOps.scala ---
@@ -186,6 +188,39 @@ class GraphOps[VD: ClassTag, ED: ClassTag](graph: 
Graph[VD, ED]) extends Seriali
   }
 
   /**
+   * Compute the neighboring vertex degree distribution.
+   *
+   * @param edgeDirection the direction along which to compute
+   * the degree distribution.
+   *
+   * @param frequencyCounter the resolution to collect the
+   * number of degrees. You can select some of pre-defined resolution 
strategies.
+   * For more details, see utils.FrequencyDistribution.
+   *
+   * @return the distribution of vertex degrees
+   */
+  def collectDegreeDist(
+  edgeDirection: EdgeDirection,
+  frequencyCounter: FrequencyDistribution = 
FrequencyDistribution.split(10))
+: Array[((Int, Int), Long)] = {
+val vertexDegrees = edgeDirection match {
+  case EdgeDirection.Either =>
+graph.vertices.leftJoin(degrees)(
+  (id, data, degree) => degree.getOrElse(0))
+  case EdgeDirection.In =>
+graph.vertices.leftJoin(inDegrees)(
+  (id, data, degree) => degree.getOrElse(0))
+  case EdgeDirection.Out =>
+graph.vertices.leftJoin(outDegrees)(
+  (id, data, degree) => degree.getOrElse(0))
--- End diff --

minor: these can fit on one line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8007#issuecomment-136919838
  
Looking better; there are still a few things that look suspicious to me in 
`ApplicationMaster`, though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2827][GraphX] Add collectDegreeDist to ...

[ankurdave]

Github user ankurdave commented on a diff in the pull request:

https://github.com/apache/spark/pull/4399#discussion_r38495393
  
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/GraphOps.scala ---
@@ -186,6 +188,39 @@ class GraphOps[VD: ClassTag, ED: ClassTag](graph: 
Graph[VD, ED]) extends Seriali
   }
 
   /**
+   * Compute the neighboring vertex degree distribution.
+   *
+   * @param edgeDirection the direction along which to compute
+   * the degree distribution.
+   *
+   * @param frequencyCounter the resolution to collect the
+   * number of degrees. You can select some of pre-defined resolution 
strategies.
+   * For more details, see utils.FrequencyDistribution.
+   *
+   * @return the distribution of vertex degrees
+   */
+  def collectDegreeDist(
+  edgeDirection: EdgeDirection,
+  frequencyCounter: FrequencyDistribution = 
FrequencyDistribution.split(10))
+: Array[((Int, Int), Long)] = {
+val vertexDegrees = edgeDirection match {
+  case EdgeDirection.Either =>
+graph.vertices.leftJoin(degrees)(
+  (id, data, degree) => degree.getOrElse(0))
+  case EdgeDirection.In =>
+graph.vertices.leftJoin(inDegrees)(
+  (id, data, degree) => degree.getOrElse(0))
+  case EdgeDirection.Out =>
+graph.vertices.leftJoin(outDegrees)(
+  (id, data, degree) => degree.getOrElse(0))
+  case _ =>
+throw new SparkException("collectDegreeDist does not support 
EdgeDirection.Both. Use" +
+  "EdgeDirection.Either instead.")
--- End diff --

Need a space in the string concatenation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7927#issuecomment-136919732
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7927#issuecomment-136919718
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38495343
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -21,8 +21,10 @@ import java.util.Collections
 import java.util.concurrent._
 import java.util.regex.Pattern
 
-import scala.collection.JavaConverters._
+import org.apache.spark.util.Utils
--- End diff --

nit: this goes with other Spark imports.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38495355
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -34,11 +36,11 @@ import org.apache.hadoop.yarn.util.RackResolver
 
 import org.apache.log4j.{Level, Logger}
 
-import org.apache.spark.{Logging, SecurityManager, SparkConf}
 import org.apache.spark.deploy.yarn.YarnSparkHadoopUtil._
 import org.apache.spark.rpc.RpcEndpointRef
-import org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages._
-import org.apache.spark.util.Utils
+import org.apache.spark.scheduler.{ExecutorExited, ExecutorLossReason}
+import 
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
+import org.apache.spark.{Logging, SecurityManager, SparkConf}
--- End diff --

nit: previous place was correct.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38495305
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -590,6 +605,18 @@ private[spark] class ApplicationMaster(
   case None => logWarning("Container allocator is not ready to 
kill executors yet.")
 }
 context.reply(true)
+
+  case GetExecutorLossReason(eid) =>
+Option(allocator) match {
+  case Some(a) =>
+pendingLossReasonRequests.synchronized {
+  pendingLossReasonRequests
--- End diff --

When would multiple requests for the same executor arrive?

Because if that case is possible, there's a possible race that would lead 
to a slow leak. If the allocator notifies of the executor reason, you'll reply 
to the existing requests; but if a new request for that executor's loss reason 
arrives after that, it will never be replied to, and forever live in 
`pendingLossReasonRequests`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38495239
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -590,6 +605,18 @@ private[spark] class ApplicationMaster(
   case None => logWarning("Container allocator is not ready to 
kill executors yet.")
 }
 context.reply(true)
+
+  case GetExecutorLossReason(eid) =>
+Option(allocator) match {
--- End diff --

Does this apply at all? You're not using `allocator` anywhere in the 
handler; so you could record the request and hope it's replied to later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38495003
  
--- Diff: core/src/test/scala/org/apache/spark/PartitioningSuite.scala ---
@@ -17,13 +17,12 @@
 
 package org.apache.spark
 
-import scala.collection.mutable.ArrayBuffer
-import scala.math.abs
-
-import org.scalatest.PrivateMethodTester
-
 import org.apache.spark.rdd.RDD
 import org.apache.spark.util.StatCounter
+import org.scalatest.PrivateMethodTester
--- End diff --

Are any of the changes in this file necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38494982
  
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -362,8 +362,9 @@ private[spark] object JsonProtocol {
 ("Stack Trace" -> stackTrace) ~
 ("Full Stack Trace" -> exceptionFailure.fullStackTrace) ~
 ("Metrics" -> metrics)
-  case ExecutorLostFailure(executorId) =>
-("Executor ID" -> executorId)
+  case ExecutorLostFailure(executorId, isNormalExit) =>
+("Executor ID" -> executorId) ~
+("Is exit reason normal" -> isNormalExit)
--- End diff --

Instead of this key I'd use something like `"Normal Exit"`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38494934
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
 ---
@@ -144,6 +209,7 @@ private[spark] abstract class YarnSchedulerBackend(
 context.reply(false)
 }
 
+  case c: GetExecutorLossReason =>
--- End diff --

Is this needed? If so, add a comment?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8007#discussion_r38494922
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
 ---
@@ -113,6 +177,7 @@ private[spark] abstract class YarnSchedulerBackend(
 removeExecutor(executorId, reason)
 }
 
+
--- End diff --

nit: not needed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/5437#issuecomment-136918531
  
@sarutak are you still actively working on this? If not, would you mind 
closing this patch for now? If there's interest in the future we can always 
reopen it against the latest master branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5719]allow daemons to bind to specified...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/4505#issuecomment-136918445
  
@WangTaoTheTonic let's close this patch since I don't think there is 
sufficient interest in this feature to warrant the additional complexity it 
introduces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org