[GitHub] spark pull request: [SPARK-10474] [SQL] Aggregation fails to alloc...

[andrewor14]

Github user andrewor14 closed the pull request at:

https://github.com/apache/spark/pull/8827


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

[yanboliang]

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/8836

[SPARK-10686] [ML] Add quantilesCol to AFTSurvivalRegression

By default ```quantilesCol``` should be empty. If 
```quantileProbabilities``` is set, we should append quantiles as a new column 
(of type Vector).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark spark-10686

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8836.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8836


commit 1d8a6828cc7078b6cba229148e9da923f8507848
Author: Yanbo Liang 
Date:   2015-09-19T06:53:43Z

Add quantilesCol to AFTSurvivalRegression




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10474] [SQL] Aggregation fails to alloc...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/8827#issuecomment-141631551
  
OK, I'm merging into master and 1.5. Thanks for the reviews.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10685] [SPARK-8632] [SQL] [PYSPARK] Pyt...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8833#discussion_r39917194
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala ---
@@ -338,7 +338,11 @@ case class BatchPythonEvaluation(udf: PythonUDF, 
output: Seq[Attribute], child:
   def children: Seq[SparkPlan] = child :: Nil
 
   protected override def doExecute(): RDD[InternalRow] = {
-val childResults = child.execute().map(_.copy())
+val buffer = new 
java.util.concurrent.ConcurrentLinkedQueue[InternalRow]()
--- End diff --

If i understand this correctly, we are assuming the following in order for 
this to work:

1. Each task gets their own copy of the deserialized closure, and thus 
their own copy of the queue.
2. All closures are serialized together in one shot, rather than in 
multiple places (e.g. they are all done in the serializer, not in the ctor of 
the RDD)
3. Java serializer does not serialize objects twice within the same stream, 
since it uses it to detect cycles. When they are deserialized, they still point 
to the same copy.







---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP] Refactor PythonRDD to decouple iterator ...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/8835

[WIP] Refactor PythonRDD to decouple iterator computation from PythonRDD.

Putting this out here for discussion with @davies.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark python-iter-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8835.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8835


commit a2474a2c2d5542a612e505fe2391e4eeb49f1bc5
Author: Reynold Xin 
Date:   2015-09-19T06:43:54Z

Refactor PythonRDD to decouple iterator computation from PythonRDD.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10631][Documentation, MLlib, PySpark]Ad...

[vinodkc]

GitHub user vinodkc opened a pull request:

https://github.com/apache/spark/pull/8834

[SPARK-10631][Documentation, MLlib, PySpark]Added documentation for few APIs

There are some missing API docs in pyspark.mllib.linalg.Vector (including 
DenseVector and SparseVector). We should add them based on their Scala 
counterparts.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vinodkc/spark fix_SPARK-10631

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8834.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8834


commit 3ee70548760c28bfc96e17a42450ac6f356de923
Author: vinodkc 
Date:   2015-09-19T06:00:26Z

Added documentation for few APIs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4131][SQL] support writing data into th...

[litao-buptsse]

Github user litao-buptsse commented on the pull request:

https://github.com/apache/spark/pull/4380#issuecomment-141625598
  
@yhuai Got it, thank you very much.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10474] [SQL] Aggregation fails to alloc...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/8827#issuecomment-141624490
  
@andrewor14 Since we will refactor all the memory management in 1.6, and 
this PR does fix the problem, so LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8632] [SQL] [PYSPARK] Poor Python UDF p...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/8662#issuecomment-141623393
  
I took the idea from @rxin to simplify CachedOnceRDD, created #8833 , 
@justinuang please help to review it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10685] [SPARK-8632] [SQL] [PYSPARK] Pyt...

[davies]

GitHub user davies opened a pull request:

https://github.com/apache/spark/pull/8833

[SPARK-10685] [SPARK-8632] [SQL] [PYSPARK] Python UDF should only compute 
the upstream once

This PR changes to buffer the rows from upstream into a Queue, then zip 
them with result from Python UDF, to avoid the double computation of upstream.

Thanks the idea from @rxin to simplify the buffer greatly!

cc @marmbrus 

Closes #8662  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davies/spark pyudf

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8833.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8833


commit b98cf2a45984d24bdb181e4d43d8f83ca1849aff
Author: Davies Liu 
Date:   2015-09-19T05:42:26Z

compute the upstream once




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141617353
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[tnachen]

Github user tnachen commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-141617352
  
I see, I can see having a minimum CPU per executor useful, that's a easy 
change in this patch, other than that i think this patch can achieve what you 
want 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10648] Proposed bug fix when oracle ret...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/8780#issuecomment-141609708
  
That should work.

On Sep 18, 2015, at 7:15 AM, Travis Hegner  wrote:

That is exactly what I was afraid of. Would the patch make more sense to
*only* check precision for a zero value? Does it ever make sense to have a
precision of zero (or less than zero for that matter)? Could we safely
enforce defaults if precision is zero (or less) regardless of scale? That
would solve my problem still, hopefully without compromising functionality
for everyone else.

—
Reply to this email directly or view it on GitHub
.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141607668
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141607670
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42699/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141607465
  
  [Test build #42699 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42699/console)
 for   PR 8832 at commit 
[`8b80886`](https://github.com/apache/spark/commit/8b80886f6be8b7df16d8d72190f3bc0b4a1611d5).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10623] [SQL] Fixes ORC predicate push-d...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8799


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141607251
  
  [Test build #42705 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42705/consoleFull)
 for   PR 8790 at commit 
[`dd615db`](https://github.com/apache/spark/commit/dd615db04da8c0f9388a90c624a303ec31df0415).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141607258
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42705/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141607257
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10258] [DOCUMENTATION, ML] Adding Since...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8505#issuecomment-141607192
  
  [Test build #42707 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42707/consoleFull)
 for   PR 8505 at commit 
[`31bf4dd`](https://github.com/apache/spark/commit/31bf4dd7323d8dc83da2557e7825f13b12efe9e7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10623] [SQL] Fixes ORC predicate push-d...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/8799#issuecomment-141607187
  
ok. Merging to master and branch 1.5.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10258] [DOCUMENTATION, ML] Adding Since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8505#issuecomment-141607033
  
 Build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10258] [DOCUMENTATION, ML] Adding Since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8505#issuecomment-141607040
  
Build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[falaki]

Github user falaki commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141606839
  
Yes, if an application starts the JVM manually (not using spark-submit) 
this call will fail. The change just adds safety without compromising 
functionality or correctness. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141606486
  
Any reason this would not be set ? My assumption was that all spark-submit 
applications had this -- so I guess this is for applications not using 
spark-submit ? 

cc @sun-rui who added the function


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9681] [ML] Support R feature interactio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8830#issuecomment-141606323
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[falaki]

Github user falaki commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141606332
  
cc @shivaram 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9681] [ML] Support R feature interactio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8830#issuecomment-141606324
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42706/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9681] [ML] Support R feature interactio...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8830#issuecomment-141606284
  
  [Test build #42706 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42706/console)
 for   PR 8830 at commit 
[`15b4da7`](https://github.com/apache/spark/commit/15b4da72a6b6bf7e0e18255dcda7a72db207ba49).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...

[Jianfeng-chs]

Github user Jianfeng-chs commented on a diff in the pull request:

https://github.com/apache/spark/pull/8801#discussion_r39914305
  
--- Diff: python/pyspark/sql/functions.py ---
@@ -189,6 +190,14 @@ def approxCountDistinct(col, rsd=None):
 return Column(jc)
 
 
+@since(1.6)
+def broadcast(df):
+"""Marks a DataFrame as small enough for use in broadcast joins."""
+
+sc = SparkContext._active_spark_context
+return DataFrame(sc._jvm.functions.broadcast(df._jdf), sc._jsc)
--- End diff --

@davies changed it. Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10623] [SQL] Fixes ORC predicate push-d...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/8799#discussion_r39913803
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -344,4 +344,34 @@ class OrcQuerySuite extends QueryTest with 
BeforeAndAfterAll with OrcTest {
   }
 }
   }
+
+  test("SPARK-10623 Enable ORC PPD") {
+withTempPath { dir =>
+  withSQLConf(SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+import testImplicits._
+
+val path = dir.getCanonicalPath
+sqlContext.range(10).coalesce(1).write.orc(path)
+val df = sqlContext.read.orc(path)
+
+def checkPredicate(pred: Column, answer: Seq[Long]): Unit = {
+  checkAnswer(df.where(pred), answer.map(Row(_)))
--- End diff --

It just checks answers. It's annoying that I couldn't find a programmatical 
way to verify whether its pushed down or not. Checked through logs manually 
though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8789#issuecomment-141603434
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9681] [ML] Support R feature interactio...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8830#issuecomment-141603326
  
  [Test build #42706 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42706/consoleFull)
 for   PR 8830 at commit 
[`15b4da7`](https://github.com/apache/spark/commit/15b4da72a6b6bf7e0e18255dcda7a72db207ba49).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8789#issuecomment-141603437
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42697/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8789#issuecomment-141602809
  
  [Test build #42697 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42697/consoleFull)
 for   PR 8789 at commit 
[`62ec0f9`](https://github.com/apache/spark/commit/62ec0f90656ad397368d9b93df721f7ec787268f).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9681] [ML] Support R feature interactio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8830#issuecomment-141602769
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9681] [ML] Support R feature interactio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8830#issuecomment-141602715
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10710] Remove ability to disable spilli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8831#issuecomment-141601050
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10710] Remove ability to disable spilli...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8831#issuecomment-141601051
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42698/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10710] Remove ability to disable spilli...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8831#issuecomment-141601028
  
  [Test build #42698 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42698/console)
 for   PR 8831 at commit 
[`bb053c7`](https://github.com/apache/spark/commit/bb053c780438848cd7fa02ab3dfb0fece1afe866).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class TaskCommitDenied(`
  * `class Interaction(override val uid: String) extends Transformer`
  * `  final val probabilityCol: Param[String] = new Param[String](this, 
"probabilityCol", "Column name for predicted class conditional probabilities. 
Note: Not all models output well-calibrated probability estimates! These 
probabilities should be treated as confidences, not precise probabilities")`
  * `class AFTSurvivalRegression @Since("1.6.0") (@Since("1.6.0") override 
val uid: String)`
  * `  require(censor == 1.0 || censor == 0.0, "censor of class AFTPoint 
must be 1.0 or 0.0")`
  * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] 
with Logging `
  * `case class Sort(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141599896
  
ok it finally loaded. So the history UI for that task reports its still 
RUNNING since it got the error parsing it.  I guess that is ok.  Ideally it 
would still show end and the task commit error even if it couldn't report the 
jod id, etc.Is that something we could do fairly easily?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141599432
  
Yeah and actually I want to do one more verification to make sure the rest 
of the history file is useful.  Unfortunately my network connection is slow 
that its a huge history file.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141599258
  
  [Test build #42705 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42705/consoleFull)
 for   PR 8790 at commit 
[`dd615db`](https://github.com/apache/spark/commit/dd615db04da8c0f9388a90c624a303ec31df0415).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141598935
  
  [Test build #42704 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42704/consoleFull)
 for   PR 8790 at commit 
[`dd615db`](https://github.com/apache/spark/commit/dd615db04da8c0f9388a90c624a303ec31df0415).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141598742
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141598731
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9962][ML][WIP] Decision Tree training: ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8541#issuecomment-141598434
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42702/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9962][ML][WIP] Decision Tree training: ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8541#issuecomment-141598432
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9962][ML][WIP] Decision Tree training: ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8541#issuecomment-141598254
  
  [Test build #42702 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42702/console)
 for   PR 8541 at commit 
[`c6e3f3f`](https://github.com/apache/spark/commit/c6e3f3f8a6838db5ffb060cc1c69f587f134ee3b).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141598171
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141598186
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141598105
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Spark-10625: Spark SQL JDBC read/write is unab...

[tribbloid]

Github user tribbloid commented on the pull request:

https://github.com/apache/spark/pull/8785#issuecomment-141598082
  
all fixed (except the PR, will fix soon), thanks a lot for pointing them 
out!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141597635
  
  [Test build #42703 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42703/consoleFull)
 for   PR 8828 at commit 
[`29001f9`](https://github.com/apache/spark/commit/29001f91d2c26e3747dc118975ce9ba95c3084b6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141596979
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42696/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141596978
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141596949
  
**[Test build #42696 timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42696/consoleFull)**
 for PR 8790 at commit 
[`dd615db`](https://github.com/apache/spark/commit/dd615db04da8c0f9388a90c624a303ec31df0415)
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10649][STREAMING] Prevent inheriting jo...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/8781#discussion_r39911678
  
--- Diff: 
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala 
---
@@ -180,6 +180,38 @@ class StreamingContextSuite extends SparkFunSuite with 
BeforeAndAfter with Timeo
 assert(ssc.scheduler.isStarted === false)
   }
 
+  test("start should set job group correctly") {
+ssc = new StreamingContext(conf, batchDuration)
+ssc.sc.setJobGroup("non-streaming", "non-streaming", true)
+val sc = ssc.sc
+
+@volatile var jobGroupFound: String = ""
+@volatile var jobDescFound: String = ""
+@volatile var jobInterruptFound: String = ""
+@volatile var allFound: Boolean = false
+
+addInputStream(ssc).foreachRDD { rdd =>
+  jobGroupFound = sc.getLocalProperty(SparkContext.SPARK_JOB_GROUP_ID)
+  jobDescFound = 
sc.getLocalProperty(SparkContext.SPARK_JOB_DESCRIPTION)
+  jobInterruptFound = 
sc.getLocalProperty(SparkContext.SPARK_JOB_INTERRUPT_ON_CANCEL)
+  allFound = true
+}
+ssc.start()
+
+eventually(timeout(5 seconds), interval(10 milliseconds)) {
+  assert(allFound === true)
+}
--- End diff --

Not necessarily. If it hangs forever Jenkins will time it out anyway. In 
general I'm not a huge fan of eventually's because these are usually the ones 
we have to end up fixing later; if you guess a timeout wrong then you'll have 
to try a slightly higher one after breaking many builds.

In this case it's probably OK to keep this as is, but I'm just saying I 
would have written the test differently.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141596522
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141596533
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141593434
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42690/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141593394
  
  [Test build #42690 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42690/console)
 for   PR 8828 at commit 
[`f144147`](https://github.com/apache/spark/commit/f14414729698ff3836688783b69c96c7fdf9361c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141593432
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10676] [docs] Add documentation for SAS...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8803#issuecomment-141593213
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10676] [docs] Add documentation for SAS...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8803#issuecomment-141593214
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42700/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10676] [docs] Add documentation for SAS...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8803#issuecomment-141593136
  
  [Test build #42700 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42700/console)
 for   PR 8803 at commit 
[`f656dfa`](https://github.com/apache/spark/commit/f656dfa367e3aeb7e54787015596532bae40a15a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-10611 Clone Configuration for each task ...

[mingyukim]

Github user mingyukim commented on the pull request:

https://github.com/apache/spark/pull/8763#issuecomment-141593054
  
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141592972
  
  [Test build #42701 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42701/consoleFull)
 for   PR 8825 at commit 
[`43eb0cf`](https://github.com/apache/spark/commit/43eb0cf2e1ee242724c6f329b8e0ce64b419b806).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] [ML] override toString of AttributeGro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8826


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-10611 Clone Configuration for each task ...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/8763#issuecomment-141592900
  
SGTM; I'll pull it in now for inclusion in 1.5.1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] [ML] override toString of AttributeGro...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/8826#issuecomment-141592830
  
Merged into master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10474] [SQL] Aggregation fails to alloc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8827#issuecomment-141592494
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10474] [SQL] Aggregation fails to alloc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8827#issuecomment-141592497
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42684/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10474] [SQL] Aggregation fails to alloc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8827#issuecomment-141592416
  
  [Test build #42684 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42684/console)
 for   PR 8827 at commit 
[`f7dd17f`](https://github.com/apache/spark/commit/f7dd17fca10d33e5afceb9df934d5a2f2290abae).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9962][ML][WIP] Decision Tree training: ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8541#issuecomment-141592371
  
  [Test build #42702 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42702/consoleFull)
 for   PR 8541 at commit 
[`c6e3f3f`](https://github.com/apache/spark/commit/c6e3f3f8a6838db5ffb060cc1c69f587f134ee3b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9962][ML][WIP] Decision Tree training: ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8541#issuecomment-141592102
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9962][ML][WIP] Decision Tree training: ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8541#issuecomment-141592089
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141592101
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141592079
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141591455
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141591264
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141591194
  
  [Test build #42691 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42691/console)
 for   PR 8825 at commit 
[`43eb0cf`](https://github.com/apache/spark/commit/43eb0cf2e1ee242724c6f329b8e0ce64b419b806).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10676] [docs] Add documentation for SAS...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8803#issuecomment-141591233
  
  [Test build #42700 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42700/consoleFull)
 for   PR 8803 at commit 
[`f656dfa`](https://github.com/apache/spark/commit/f656dfa367e3aeb7e54787015596532bae40a15a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141591265
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42691/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10676] [docs] Add documentation for SAS...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8803#issuecomment-141591077
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10676] [docs] Add documentation for SAS...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8803#issuecomment-141591062
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141590625
  
  [Test build #42699 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42699/consoleFull)
 for   PR 8832 at commit 
[`8b80886`](https://github.com/apache/spark/commit/8b80886f6be8b7df16d8d72190f3bc0b4a1611d5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

[SleepyThread]

Github user SleepyThread commented on the pull request:

https://github.com/apache/spark/pull/4027#issuecomment-141590430
  
There is no minimum core per executor cap in this patch, only maximum 
cores. Also i think we will need to add more documentation around it to make 
these parameter more clear with examples [I can add them once pull request is 
accepted].


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141589747
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/8789#issuecomment-141589791
  
LGTM. We can merge it once jenkins is good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8832#issuecomment-141589739
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/8790#issuecomment-141589683
  
LGTM. We can merge it once jenkins is good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARKR][SPARK-10711] Do not assume spark.subm...

[falaki]

GitHub user falaki opened a pull request:

https://github.com/apache/spark/pull/8832

[SPARKR][SPARK-10711] Do not assume spark.submit.deployMode is always set

In ```RUtils.sparkRPackagePath()``` we 
1. Call ``` sys.props("spark.submit.deployMode")``` which returns null if 
```spark.submit.deployMode``` is not suet
2. Call ``` sparkConf.get("spark.submit.deployMode")``` which throws 
```NoSuchElementException``` if ```spark.submit.deployMode``` is not set. This 
patch simply passes a default value ("cluster") for 
```spark.submit.deployMode```.

cc @rxin 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/falaki/spark SPARK-10711

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8832.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8832


commit 854e5abb2aee9302755fd119580a0e437be008f0
Author: Hossein 
Date:   2015-09-18T22:53:19Z

Assume a default value for spark.submit.deployMode in RUtils

commit 8b80886f6be8b7df16d8d72190f3bc0b4a1611d5
Author: Hossein 
Date:   2015-09-18T23:00:25Z

Changed default value for spark.submit.deployMode to cluster




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

reviews@spark.apache.org

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8790#discussion_r39909311
  
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala ---
@@ -29,7 +29,7 @@ import org.apache.spark.annotation.DeveloperApi
 class TaskInfo(
 val taskId: Long,
 val index: Int,
-val attempt: Int,
--- End diff --

Maybe we need to take a look at the code to see the meaning of `attempt`.  
I find 
https://github.com/apache/spark/blob/branch-1.3/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L451-L452.
 Looks like it is indeed `attemptNumber`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10640] History server fails to parse Ta...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/8828#issuecomment-141588955
  
I see, because in your event log we only logged the name `TaskCommitDenied` 
but not the job ID and other details. That makes sense.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10250][CORE] External group by to handl...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/8438#issuecomment-141588625
  
Hey @mccheah,

This is a really cool patch. Although we're trying to encourage users to 
migrate workloads to the DataFrames API, there are still many workloads for 
which this is a useful improvement. This patch has some high inherent 
complexity and risk, though, since the details of managing file cleanup are 
non-trivial and it needs to touch a number of stable code paths which haven't 
been modified in a long time.

I'd like to review this patch but I'm a bit too busy with other 1.6 
development and design tasks to devote adequate review time right now. I may 
have some time to review this in a couple of weeks, though. If you'd like to 
try to make this easier to review, I have a couple of suggestions (perhaps 
off-base, since I haven't considered them in detail yet):

1. Split the ContextCleaner changes into their own JIRA + PR.
2. Guard the external groupBy behind a feature-flag so that users have an 
escape-hatch in case bugs are uncovered in the new external groupBy 
implementation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141588199
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42682/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141588197
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10704] Consolidate HashShuffleReader an...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8825#issuecomment-141588101
  
  [Test build #42682 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42682/console)
 for   PR 8825 at commit 
[`36b25e3`](https://github.com/apache/spark/commit/36b25e347f0573d96a5940ead837e02a10e5ae08).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org