[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on a diff in the pull request:

https://github.com/apache/spark/pull/9837#discussion_r45450002
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
   " specified in spark.yarn.keytab does not exist")
   } else {
 logInfo("Attempting to login to Kerberos" +
-  s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --

Hi!
I think it is required because of another context of execution in this part 
of code. Here is a context of the classloader, which cause an empty 
environment, with default value of authentication, which is "simple". In case 
of "simple" a keytab is not used and credentials cannot be granted. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11818][REPL] Fix ExecutorClassLoader to...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/spark/pull/9812#issuecomment-158338205
  
@vanzin Thanks for reviewing, I addressed your comment. Please take a look 
again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11753][SQL] Make allowNonNumericNumbers...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9759#issuecomment-158348589
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11753][SQL] Make allowNonNumericNumbers...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9759#issuecomment-158348592
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46407/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11753][SQL] Make allowNonNumericNumbers...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9759#issuecomment-158348538
  
**[Test build #46407 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46407/consoleFull)**
 for PR 9759 at commit 
[`186fa5e`](https://github.com/apache/spark/commit/186fa5e2f9bcc8c66bbe51c5d03e3a8f681d051a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11818][REPL] Fix ExecutorClassLoader to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9812#issuecomment-158369660
  
**[Test build #46410 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46410/consoleFull)**
 for PR 9812 at commit 
[`ac32668`](https://github.com/apache/spark/commit/ac326687716f842cf1fb5e165bfc10fcf6fea6fd).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158373123
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46415/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158373120
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158377704
  
**[Test build #46417 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46417/consoleFull)**
 for PR 6263 at commit 
[`33fc892`](https://github.com/apache/spark/commit/33fc892c84a0a0ef7375c26eba1e581ffc108a2b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11818][REPL] Fix ExecutorClassLoader to...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/spark/pull/9812#issuecomment-158377559
  
Failed tests seems not related.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158385311
  
**[Test build #46418 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46418/consoleFull)**
 for PR 6263 at commit 
[`2cf4f71`](https://github.com/apache/spark/commit/2cf4f718d7a72fbc5fb663d5576bb7de88bfdeff).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158385324
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46418/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158385319
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-158413781
  
**[Test build #46421 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46421/consoleFull)**
 for PR 8880 at commit 
[`34b0578`](https://github.com/apache/spark/commit/34b057861907f2262bc6ba55f1accc17ef93e0f3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158425982
  
**[Test build #46422 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46422/consoleFull)**
 for PR 6263 at commit 
[`e1577dc`](https://github.com/apache/spark/commit/e1577dc1b364aa3a4e923ebe3ddc0e82d270cef5).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158425985
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5106 - Add refresh checkbox on the UI

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/9860#issuecomment-158425481
  
Wont be of much merit on the completed history server UIs, so ought to be 
something which can be completely omitted off the page


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158425439
  
**[Test build #46422 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46422/consoleFull)**
 for PR 6263 at commit 
[`e1577dc`](https://github.com/apache/spark/commit/e1577dc1b364aa3a4e923ebe3ddc0e82d270cef5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6791] [ML] Add read/write for CrossVali...

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/9848#discussion_r45468030
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala ---
@@ -18,17 +18,23 @@
 package org.apache.spark.ml.tuning
 
 import com.github.fommil.netlib.F2jBLAS
+import org.apache.hadoop.fs.Path
+import org.json4s.JsonAST.{JString, JNothing, JObject}
+import org.json4s.{JValue, DefaultFormats}
+import org.json4s.jackson.JsonMethods._
 
-import org.apache.spark.Logging
-import org.apache.spark.annotation.Experimental
+import org.apache.spark.ml.util.DefaultParamsReader.Metadata
+import org.apache.spark.{SparkContext, Logging}
+import org.apache.spark.annotation.{Experimental, Since}
--- End diff --

organize imports


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158425617
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46420/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11881][SQL] Fix for postgresql fetchsiz...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9861#issuecomment-158417791
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158425988
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46422/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6791] [ML] Add read/write for CrossVali...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/9848#issuecomment-158406894
  
LGTM for me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11881][SQL] Fix for postgresql fetchsiz...

[mariusvniekerk]

GitHub user mariusvniekerk opened a pull request:

https://github.com/apache/spark/pull/9861

[SPARK-11881][SQL] Fix for postgresql fetchsize > 0

Reference: 
https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor
In order for PostgreSQL to honor the fetchSize non-zero setting, its 
Connection.autoCommit needs to be set to false. Otherwise, it will just quietly 
ignore the fetchSize setting.

This adds a new side-effecting dialect specific beforeFetch method that 
will fire before a select query is ran.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariusvniekerk/spark SPARK-11881

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9861.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9861


commit 464976670c2857a972b38aa4d32396915d7e0c0a
Author: mariusvniekerk 
Date:   2015-11-20T14:34:31Z

[SPARK-11881][SQL] Fix for postgresql fetchsize > 0

Reference: 
https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor
In order for PostgreSQL to honor the fetchSize non-zero setting, its 
Connection.autoCommit needs to be set to false. Otherwise, it will just quietly 
ignore the fetchSize setting.

This adds a new side-effecting dialect specific beforeFetch method that 
will fire before a select query is ran.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158425614
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158425456
  
**[Test build #46420 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46420/consoleFull)**
 for PR 9736 at commit 
[`011e674`](https://github.com/apache/spark/commit/011e674177fb60400289fa4043dc9633c3a815c1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11863][SQL][WIP] Unable to resolve orde...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9844#issuecomment-158351303
  
**[Test build #46405 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46405/consoleFull)**
 for PR 9844 at commit 
[`ef4274a`](https://github.com/apache/spark/commit/ef4274a06a5859e0f233baaab4044a6d1b0e6418).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11863][SQL][WIP] Unable to resolve orde...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9844#issuecomment-158351431
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46405/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158357514
  
**[Test build #46414 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46414/consoleFull)**
 for PR 6263 at commit 
[`3e23321`](https://github.com/apache/spark/commit/3e233216df375345ad7e998be9567f3acc903110).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10387][ML][WIP] Add code gen for gbt

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9524#issuecomment-158359278
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158359476
  
**[Test build #46415 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46415/consoleFull)**
 for PR 9839 at commit 
[`c6b6d7e`](https://github.com/apache/spark/commit/c6b6d7e65d96b681ecaa6ac004f8637f72bc0802).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[archit279thakur]

Github user archit279thakur commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158366439
  
@squito 
In reply to:
*@archit279thakur can you also bring this up to date with master, and 
include before & after screenshots? I'd like for this to also update the json 
endpoints. Finally, I think that as long we're storing removed executors, we 
should store the time they were removed.*

Two things:
1. For that, I'll have to add a new column in the execTable on the UI. 
Should it be lastStatusChangedTime (with aliveTime or killedTime, depending on 
the status) or KilledTime (with blank values for the Alive executors) ?
2. This value would always be greater than the value currentTime - 
spark.ui.timeToExpireKilledExecutor. I am not really sure, we provide any 
useful insights by showing the time at which executor died.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5106 - Add refresh checkbox on the UI

[jbonofre]

Github user jbonofre commented on the pull request:

https://github.com/apache/spark/pull/9860#issuecomment-158377000
  
Thanks for the update Sean. Some background:
1. I wanted to avoid "intrusive" code on the server side, that's why the 
refresh is "client side" using JavaScript.
2. I disable the auto-refresh by default, and let the user enable it if 
wanted.
3. I think it's useful (for instance, a simple case is when you are on the 
master UI, and just want to see running applications, workers, etc status 
change)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158386948
  
**[Test build #46419 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46419/consoleFull)**
 for PR 9736 at commit 
[`0291a30`](https://github.com/apache/spark/commit/0291a3037ed18b7a144fe7f9e27c3b209a314f0a).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158386950
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11856][SQL][WIP] add type cast if the r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9840#issuecomment-158337182
  
**[Test build #46409 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46409/consoleFull)**
 for PR 9840 at commit 
[`2197018`](https://github.com/apache/spark/commit/21970181912043c1cb4057c1553c4bc1992d7826).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11818][REPL] Fix ExecutorClassLoader to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9812#issuecomment-158338829
  
**[Test build #46410 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46410/consoleFull)**
 for PR 9812 at commit 
[`ac32668`](https://github.com/apache/spark/commit/ac326687716f842cf1fb5e165bfc10fcf6fea6fd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158353678
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158353682
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46412/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158368335
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46413/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158368333
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11818][REPL] Fix ExecutorClassLoader to...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/9812#discussion_r45451124
  
--- Diff: 
repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala ---
@@ -55,6 +57,14 @@ class ExecutorClassLoader(conf: SparkConf, classUri: 
String, parent: ClassLoader
 }
   }
 
+  override def getResource(name: String): URL = {
+parentLoader.getResource(name)
--- End diff --

That all seams reasonable to me (sorry for the slow reply I'm in Tokyo 
right now).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11856][SQL][WIP] add type cast if the r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9840#issuecomment-158344436
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46409/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11856][SQL][WIP] add type cast if the r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9840#issuecomment-158344392
  
**[Test build #46409 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46409/consoleFull)**
 for PR 9840 at commit 
[`2197018`](https://github.com/apache/spark/commit/21970181912043c1cb4057c1553c4bc1992d7826).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11856][SQL][WIP] add type cast if the r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9840#issuecomment-158344433
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10387][ML][WIP] Add code gen for gbt

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9524#issuecomment-158346630
  
**[Test build #46411 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46411/consoleFull)**
 for PR 9524 at commit 
[`b0129fa`](https://github.com/apache/spark/commit/b0129fab13f1d471c4ffc41e7fda2a2492cc281e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11863][SQL][WIP] Unable to resolve orde...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9844#issuecomment-158351430
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11636][SQL] Support classes defined in ...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/9825#discussion_r45457154
  
--- Diff: 
repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkIMain.scala ---
@@ -1221,10 +1221,16 @@ import org.apache.spark.annotation.DeveloperApi
 )
   }
 
-  val preamble = """
-|class %s extends Serializable {
-|  %s%s%s
-  """.stripMargin.format(lineRep.readName, envLines.map("  " + _ + 
";\n").mkString, importsPreamble, indentCode(toCompute))
+  val preamble = s"""
+|class ${lineRep.readName} extends Serializable {
+|  ${envLines.map("  " + _ + ";\n").mkString}
+|  $importsPreamble
+|
+|  // If we need to construct any objects defined in the REPL on 
an executor we will need
+|  // to pass the outer scope to the appropriate encoder.
+|  
org.apache.spark.sql.catalyst.encoders.OuterScopes.addOuterScope(this)
--- End diff --

So in 2.11 you probably don't have this problem, do you? All enclosing 
entities are objects, and they don't need an outer pointer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158357990
  
**[Test build #46414 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46414/consoleFull)**
 for PR 6263 at commit 
[`3e23321`](https://github.com/apache/spark/commit/3e233216df375345ad7e998be9567f3acc903110).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158357994
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46414/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158357992
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11818][REPL] Fix ExecutorClassLoader to...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9812#issuecomment-158369725
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46410/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158369809
  
**[Test build #46416 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46416/consoleFull)**
 for PR 6263 at commit 
[`b827f8f`](https://github.com/apache/spark/commit/b827f8f0c8f463cc181608e927add71a2cae71b0).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11818][REPL] Fix ExecutorClassLoader to...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9812#issuecomment-158369724
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158369893
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46416/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158369890
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7857][MLLIB] Prevent IDFModel from retu...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9843#discussion_r45461570
  
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala ---
@@ -218,7 +218,7 @@ private object IDFModel {
   newValues(k) = values(k) * idf(indices(k))
   k += 1
 }
-Vectors.sparse(n, indices, newValues)
+Vectors.sparse(n, indices, newValues).toSparse
--- End diff --

Hm, isn't this an optimization to make straight in the `Vectors.sparse` 
method? it doesn't make sense that you have to call `toSparse` on its output, 
that that would have any effect.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158385763
  
**[Test build #46419 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46419/consoleFull)**
 for PR 9736 at commit 
[`0291a30`](https://github.com/apache/spark/commit/0291a3037ed18b7a144fe7f9e27c3b209a314f0a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

GitHub user woj-i opened a pull request:

https://github.com/apache/spark/pull/9859

[SPARK-11821] Propagate Kerberos keytab for all environments

@andrewor14 the same PR as in branch 1.5
@harishreedharan 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/woj-i/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9859.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9859


commit 0b55df5c433cd1052793e03eea4022b3f3b5f94b
Author: woj-i 
Date:   2015-11-18T18:14:31Z

[SPARK-11821] location of Kerberos keytab in a property for all 
environments, bugfix of renewing credentials in the Classloader context

commit b3b0d59bfb92b86434d673280e43ce1ba3c6c080
Author: woj-i 
Date:   2015-11-20T09:51:05Z

[SPARK-11821] bugfix of renewing Kerberos credentials in context of the 
classloader




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158355596
  
**[Test build #46413 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46413/consoleFull)**
 for PR 9839 at commit 
[`c6b6d7e`](https://github.com/apache/spark/commit/c6b6d7e65d96b681ecaa6ac004f8637f72bc0802).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158368130
  
**[Test build #46413 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46413/consoleFull)**
 for PR 9839 at commit 
[`c6b6d7e`](https://github.com/apache/spark/commit/c6b6d7e65d96b681ecaa6ac004f8637f72bc0802).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158386953
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46419/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11636][SQL] Support classes defined in ...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/9825#discussion_r45450219
  
--- Diff: 
repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkIMain.scala ---
@@ -1221,10 +1221,16 @@ import org.apache.spark.annotation.DeveloperApi
 )
   }
 
-  val preamble = """
-|class %s extends Serializable {
-|  %s%s%s
-  """.stripMargin.format(lineRep.readName, envLines.map("  " + _ + 
";\n").mkString, importsPreamble, indentCode(toCompute))
+  val preamble = s"""
+|class ${lineRep.readName} extends Serializable {
+|  ${envLines.map("  " + _ + ";\n").mkString}
+|  $importsPreamble
+|
+|  // If we need to construct any objects defined in the REPL on 
an executor we will need
+|  // to pass the outer scope to the appropriate encoder.
+|  
org.apache.spark.sql.catalyst.encoders.OuterScopes.addOuterScope(this)
--- End diff --

The REPL in 2.11 is pretty much the Scala standard REPL:


https://github.com/apache/spark/tree/master/repl/scala-2.11/src/main/scala/org/apache/spark/repl


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10387][ML][WIP] Add code gen for gbt

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9524#issuecomment-158358994
  
**[Test build #46411 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46411/consoleFull)**
 for PR 9524 at commit 
[`b0129fa`](https://github.com/apache/spark/commit/b0129fab13f1d471c4ffc41e7fda2a2492cc281e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * `
trait CallableVectorDouble `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10387][ML][WIP] Add code gen for gbt

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9524#issuecomment-158359281
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46411/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5106 - Add refresh checkbox on the UI

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9860#issuecomment-158375239
  
I kind of like the functionality, myself. However I'm only concerned about 
where and how to expose this. I want to avoid tacking on bits and pieces to the 
UI until it's just a quilt of info and links. This seems pretty reasonable, 
still.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158378393
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46417/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158378387
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158378375
  
**[Test build #46417 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46417/consoleFull)**
 for PR 6263 at commit 
[`33fc892`](https://github.com/apache/spark/commit/33fc892c84a0a0ef7375c26eba1e581ffc108a2b).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158382654
  
**[Test build #46418 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46418/consoleFull)**
 for PR 6263 at commit 
[`2cf4f71`](https://github.com/apache/spark/commit/2cf4f718d7a72fbc5fb663d5576bb7de88bfdeff).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11636][SQL] Support classes defined in ...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/9825#discussion_r45456679
  
--- Diff: 
repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkIMain.scala ---
@@ -1221,10 +1221,16 @@ import org.apache.spark.annotation.DeveloperApi
 )
   }
 
-  val preamble = """
-|class %s extends Serializable {
-|  %s%s%s
-  """.stripMargin.format(lineRep.readName, envLines.map("  " + _ + 
";\n").mkString, importsPreamble, indentCode(toCompute))
+  val preamble = s"""
+|class ${lineRep.readName} extends Serializable {
+|  ${envLines.map("  " + _ + ";\n").mkString}
+|  $importsPreamble
+|
+|  // If we need to construct any objects defined in the REPL on 
an executor we will need
+|  // to pass the outer scope to the appropriate encoder.
+|  
org.apache.spark.sql.catalyst.encoders.OuterScopes.addOuterScope(this)
--- End diff --

Yes @dragos, that is why I am concerned.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11636][SQL] Support classes defined in ...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/9825#discussion_r45456618
  
--- Diff: 
repl/scala-2.10/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -278,6 +281,27 @@ class ReplSuite extends SparkFunSuite {
 assertDoesNotContain("java.lang.ClassNotFoundException", output)
   }
 
+  test("Datasets and encoders") {
+val output = runInterpreter("local",
+  """
+|import org.apache.spark.sql.functions._
+|import org.apache.spark.sql.Encoder
+|import org.apache.spark.sql.expressions.Aggregator
+|import org.apache.spark.sql.TypedColumn
+|val simpleSum = new Aggregator[Int, Int, Int] with Serializable {
+|  def zero: Int = 0 // The initial value.
+|  def reduce(b: Int, a: Int) = b + a// Add an element to the 
running total
+|  def merge(b1: Int, b2: Int) = b1 + b2 // Merge intermediate 
values.
+|  def finish(b: Int) = b// Return the final 
result.
+|}.toColumn
+|
+|val ds = Seq(1, 2, 3, 4).toDS()
+|ds.select(simpleSum).collect
+  """.stripMargin)
+assertDoesNotContain("error:", output)
+assertDoesNotContain("Exception", output)
--- End diff --

Yes, I can see it failing now both in scala 2.10 and 2.11.

I will spend more time to understand, if it is possible to fix it in some 
other way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158366325
  
**[Test build #46416 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46416/consoleFull)**
 for PR 6263 at commit 
[`b827f8f`](https://github.com/apache/spark/commit/b827f8f0c8f463cc181608e927add71a2cae71b0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5106 - Add refresh checkbox on the UI

[jbonofre]

Github user jbonofre commented on the pull request:

https://github.com/apache/spark/pull/9860#issuecomment-158373980
  
Rebase


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5106 - Add refresh checkbox on the UI

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/spark/pull/9860

SPARK-5106 - Add refresh checkbox on the UI

Add a checkbox to enable auto-refresh. See attached screenshots:


![screen](https://cloud.githubusercontent.com/assets/158903/11299227/590d87b6-8f85-11e5-8706-805383e7e5fb.png)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/spark SPARK-5106

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9860.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9860


commit ac19324d35b1b4fd74ad07fc0c9d9718e14a30da
Author: Jean-Baptiste Onofré 
Date:   2015-11-20T11:50:00Z

SPARK-5106 - Add refresh checkbox on the UI




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-5106 - Add refresh checkbox on the UI

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9860#issuecomment-158373998
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158355730
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11689] [ML] Add user guide and example ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9722#issuecomment-158336112
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11689] [ML] Add user guide and example ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9722#issuecomment-158335989
  
**[Test build #46408 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46408/consoleFull)**
 for PR 9722 at commit 
[`c794096`](https://github.com/apache/spark/commit/c794096d7ae52a91b577640d27f480860f4b31d0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`public class JavaLDAExample `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11689] [ML] Add user guide and example ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9722#issuecomment-158336115
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46408/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10387][ML][WIP] Add code gen for gbt

[holdenk]

GitHub user holdenk reopened a pull request:

https://github.com/apache/spark/pull/9524

[SPARK-10387][ML][WIP] Add code gen for gbt

This PR adds code gen for GBT in ML. It does this using quasi quotes since 
they give nice compile time type messages. The API could also use some feedback 
for how we want to expose code gen - the current iteration gives the user a 
`toCodeGen` function they can call, but we could also use a config flag or 
something similar (open to suggestions). This is based on DB's work (as 
mentioned in the Jira)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/spark SPARK-10387-code-gen-for-gbt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9524.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9524


commit de332b7cebe2e44dda46ad95214479973380f9b3
Author: Holden Karau 
Date:   2015-11-02T20:41:06Z

Start some work towards adding a codegen capable model

commit 344349a28b40b9e9c3b9026f2b0dbd4704567358
Author: Holden Karau 
Date:   2015-11-03T20:12:48Z

Merge branch 'master' into SPARK-10387-code-gen-for-gbt

commit 8be52cc0a4650164ccea2285e23473e7160d3156
Author: Holden Karau 
Date:   2015-11-04T20:08:47Z

Try and get quasi quotes for 2.10

commit 7ec3b4a78c667db43211b08a8ed0edf9dc5d968e
Author: Holden Karau 
Date:   2015-11-04T20:43:25Z

move tree bits around some

commit 28c134f29bc2b244b210f314324245f1fe25c660
Author: Holden Karau 
Date:   2015-11-04T21:33:24Z

Add quasiquotes lib

commit df279367a3b1772df285fd9201fd12303a9d69e5
Author: Holden Karau 
Date:   2015-11-04T21:36:54Z

Update based on what spark sql used to do

commit 3c3e67e1427a3b22c138e954c4c03247402a42b8
Author: Holden Karau 
Date:   2015-11-04T21:37:05Z

Update tree model codegen a bit

commit 3d8a6ad08654b3057d069e871802d7ad2165f185
Author: Holden Karau 
Date:   2015-11-04T21:54:29Z

Well Spark SQL put it in the root pom, maybe for a good reason. Lets try 
that

commit 0ec21f7fbbb1d10ad4cf38499f608b7727a0c878
Author: Holden Karau 
Date:   2015-11-04T23:31:30Z

It compiles

commit 5cfbf5c024034c93f8c213255e80aa3cd7e4912e
Author: Holden Karau 
Date:   2015-11-05T00:47:15Z

It compiles

commit d2b90066bd1f3bc1b82dff54d4d8084c93f21962
Author: Holden Karau 
Date:   2015-11-05T01:06:23Z

Cleanup some whitespace

commit 64a69104b1b35b2f3289a82918dc2916abd9cdb0
Author: Holden Karau 
Date:   2015-11-05T19:11:56Z

Add missing scala-compiler depedency so that quasi quotes can do its thing

commit 204fa7c6ecb5fb393d353978158191c7c60afda6
Author: Holden Karau 
Date:   2015-11-05T19:17:31Z

tests pass! #shipit

commit a1eadf0198c630cc192a13c4b475a45644e32eb6
Author: Holden Karau 
Date:   2015-11-05T19:40:31Z

Style fixes

commit a2de15b5411acfab0382479ce508a6146bf302f3
Author: Holden Karau 
Date:   2015-11-06T07:48:53Z

Merge branch 'master' into SPARK-10387-code-gen-for-gbt

commit de077419c3f9fb8de4d78d8adfaf039a992f8328
Author: Holden Karau 
Date:   2015-11-06T16:26:10Z

Add an explicit leaf node test

commit 037e611965bcb0ac9d5b9b2ab6936a1e6b5d3c0c
Author: Holden Karau 
Date:   2015-11-06T16:58:06Z

Use toCodeGen instead of just auto doing it for trees under 400, add it for 
GBT classifier as well

commit 70b00dbd44022af6cbc0718560eb2ecfb7de39e6
Author: Holden Karau 
Date:   2015-11-06T18:11:16Z

Add explicit tests for a tree conversion to codegen

commit b27e5531dc573afa5ca99f68a61c1e69f0267b53
Author: Holden Karau 
Date:   2015-11-06T18:13:01Z

whoops need to be ge not geq for cond

commit aa315e202cebecf8de041358047919f0c5ac08a5
Author: Holden Karau 
Date:   2015-11-06T21:31:43Z

rip out quasi quotes for mllib

commit ca72c1c27a9d81d048019a3cc687cb65595ea516
Author: Holden Karau 
Date:   2015-11-06T21:54:00Z

Start switching MLLib codejen to java codegen

commit 9bc39bfaaa67656272df5723596c2fc5c42e79f3
Author: Holden Karau 
Date:   2015-11-06T22:34:48Z

Add missing profiles close tag

commit fa6764e98da0bac01400c2e1dca5b6a245fddf90
Author: Holden Karau 
Date:   2015-11-07T06:09:58Z

move the test

commit f12d0bd4708d4fde79c43439b701a176843e6cfa
Author: Holden Karau 
Date:   2015-11-07T07:38:19Z

Ok seems to run now

commit 471b934673d199f6649ae4f8dcaad68c1fb22313
Author: Holden Karau 
Date:   2015-11-08T03:12:51Z

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-158344013
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10387][ML][WIP] Add code gen for gbt

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/9524#issuecomment-158343945
  
@dbtsai : I've fixed the issue with generating large trees (depth > 10) 
that I found while doing the performance testing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-158349971
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46404/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-158349801
  
**[Test build #46404 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46404/consoleFull)**
 for PR 9858 at commit 
[`3ec2dbb`](https://github.com/apache/spark/commit/3ec2dbb950546477928cb4dcb2812cc370491b8b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * `  
class ChiSqSelectorModelWriter(instance: ChiSqSelectorModel) extends MLWriter 
`\n  * `class PCA (override val uid: String) extends Estimator[PCAModel] with 
PCAParams`\n  * `  class VectorIndexerModelWriter(instance: VectorIndexerModel) 
extends MLWriter `\n  * `final class Word2Vec(override val uid: String) extends 
Estimator[Word2VecModel] with Word2VecBase`\n  * `  class 
Word2VecModelWriter(instance: Word2VecModel) extends MLWriter `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-158349969
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11852] [ML] StandardScaler minor refact...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9839#issuecomment-158372744
  
**[Test build #46415 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46415/consoleFull)**
 for PR 9839 at commit 
[`c6b6d7e`](https://github.com/apache/spark/commit/c6b6d7e65d96b681ecaa6ac004f8637f72bc0802).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-158398707
  
**[Test build #46420 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46420/consoleFull)**
 for PR 9736 at commit 
[`011e674`](https://github.com/apache/spark/commit/011e674177fb60400289fa4043dc9633c3a815c1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158435009
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46423/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7857][MLLIB] Prevent IDFModel from retu...

[karlhigley]

Github user karlhigley commented on a diff in the pull request:

https://github.com/apache/spark/pull/9843#discussion_r45481318
  
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala ---
@@ -218,7 +218,7 @@ private object IDFModel {
   newValues(k) = values(k) * idf(indices(k))
   k += 1
 }
-Vectors.sparse(n, indices, newValues)
+Vectors.sparse(n, indices, newValues).toSparse
--- End diff --

I tend to agree that this way feels strange. This was the suggested 
solution in the JIRA ticket, but I'd be happy to update `Vectors.sparse` 
instead. I considered it, but wasn't sure it made sense to strip out explicit 
zeros unless it was actually needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158435008
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158430559
  
**[Test build #46423 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46423/consoleFull)**
 for PR 6263 at commit 
[`826587f`](https://github.com/apache/spark/commit/826587fab9e2976092c0ed54b385d4c8cb596fbf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-158449167
  
**[Test build #46421 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46421/consoleFull)**
 for PR 8880 at commit 
[`34b0578`](https://github.com/apache/spark/commit/34b057861907f2262bc6ba55f1accc17ef93e0f3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-158449433
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46421/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-158449428
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-7729:Executor which has been killed shou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6263#issuecomment-158434952
  
**[Test build #46423 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46423/consoleFull)**
 for PR 6263 at commit 
[`826587f`](https://github.com/apache/spark/commit/826587fab9e2976092c0ed54b385d4c8cb596fbf).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7857][MLLIB] Prevent IDFModel from retu...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9843#discussion_r45484200
  
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala ---
@@ -218,7 +218,7 @@ private object IDFModel {
   newValues(k) = values(k) * idf(indices(k))
   k += 1
 }
-Vectors.sparse(n, indices, newValues)
+Vectors.sparse(n, indices, newValues).toSparse
--- End diff --

Oh, hm, maybe the idea is that checking this condition and 'fixing it' in 
the constructor is too costly. @jkbradley do I have the logic right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11872] Prevent the call to SparkContext...

[tedyu]

Github user tedyu commented on the pull request:

https://github.com/apache/spark/pull/9852#issuecomment-158465022
  
@andrewor14 @zsxwing 
Kindly review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on a diff in the pull request:

https://github.com/apache/spark/pull/9837#discussion_r45495036
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
   " specified in spark.yarn.keytab does not exist")
   } else {
 logInfo("Attempting to login to Kerberos" +
-  s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --

Yes, I did.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10180] [SQL] JDBC datasource are not pr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8743#issuecomment-158482749
  
**[Test build #46425 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46425/consoleFull)**
 for PR 8743 at commit 
[`bf0b22c`](https://github.com/apache/spark/commit/bf0b22c75b1f5084a07daa903d554dafc20d9a6a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org