[GitHub] spark pull request: [SPARK-11684] [R] [ML] [Doc] Update SparkR glm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9727#issuecomment-157334398
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46077/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11684] [R] [ML] [Doc] Update SparkR glm...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9727#issuecomment-157334049
  
**[Test build #46077 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46077/consoleFull)**
 for PR 9727 at commit 
[`dc7edf6`](https://github.com/apache/spark/commit/dc7edf6d5098ee36bda2e467dc2b94cd68774758).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157337188
  
**[Test build #2074 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2074/consoleFull)**
 for PR 9165 at commit 
[`9a79862`](https://github.com/apache/spark/commit/9a79862be3e7aa8ec4cf6e57e0e826b211f81e27).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11195][CORE] Use correct classloader fo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9367#issuecomment-157336863
  
  [Test build #2073 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2073/consoleFull)
 for   PR 9367 at commit 
[`fadf2ca`](https://github.com/apache/spark/commit/fadf2ca57e21d5f2600d4ec3e504e20db596c35c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11016] Move RoaringBitmap to explicit K...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/9748#discussion_r45046886
  
--- Diff: 
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -377,6 +374,55 @@ private[serializer] object KryoSerializer {
 classOf[BoundedPriorityQueue[_]],
 classOf[SparkConf]
   )
+
+  private val toRegisterSerializer = Map[Class[_], KryoClassSerializer[_]](
+classOf[RoaringBitmap] -> new KryoClassSerializer[RoaringBitmap](){
--- End diff --

Space before `{`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11016] Move RoaringBitmap to explicit K...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/9748#discussion_r45046890
  
--- Diff: 
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -377,6 +374,55 @@ private[serializer] object KryoSerializer {
 classOf[BoundedPriorityQueue[_]],
 classOf[SparkConf]
   )
+
+  private val toRegisterSerializer = Map[Class[_], KryoClassSerializer[_]](
+classOf[RoaringBitmap] -> new KryoClassSerializer[RoaringBitmap](){
+  override def write(kryo: Kryo, output: KryoOutput, bitmap: 
RoaringBitmap): Unit =
+   bitmap.serialize(new KryoOutputDataOutputBridge(output))
--- End diff --

Indentation is off.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11016] Move RoaringBitmap to explicit K...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9748#issuecomment-157340731
  
Oh I didn't notice that #9243 was reverted. So `RoaringBitmap` is still a 
direct dependency, and before this PR, Kryo can't serialize it properly. Then 
this LGTM except for a few minor styling issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9165#discussion_r45052442
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/sorttable.js 
---
@@ -169,7 +169,7 @@ sorttable = {
 for (var i=0; i

[GitHub] spark pull request: [SPARK-11751]Doc describe error in the "Spark ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9734#issuecomment-157360669
  
**[Test build #2077 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2077/consoleFull)**
 for PR 9734 at commit 
[`afd1dbc`](https://github.com/apache/spark/commit/afd1dbc7e45e49f0f38d58968ad57f0a670fb008).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11773][SPARKR] Implement collection fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9764#issuecomment-157362476
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46079/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11773][SPARKR] Implement collection fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9764#issuecomment-157362475
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157362455
  
**[Test build #46081 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46081/consoleFull)**
 for PR 9740 at commit 
[`8b70477`](https://github.com/apache/spark/commit/8b70477048208b41ba03b524abc510191633e5e0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157362569
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46081/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11694] [SQL] Backports #9754

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9763#issuecomment-157371547
  
**[Test build #46082 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46082/consoleFull)**
 for PR 9763 at commit 
[`bedb89a`](https://github.com/apache/spark/commit/bedb89a28acd88251c418b113bcb431bfe676ccc).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7888#issuecomment-157335123
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7888#issuecomment-157335041
  
**[Test build #46073 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46073/consoleFull)**
 for PR 7888 at commit 
[`1938e61`](https://github.com/apache/spark/commit/1938e6158838bb1f21884c3d3eba9f0b8ffabe05).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11698][Streaming] Add option to ignore ...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/9665#issuecomment-157335148
  
This seems like a very very specific niche use case. I am not sure whether 
this sort of a one-off option is a good idea. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11779][DOCS] Fix reference to deprecate...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9768#issuecomment-157346906
  
**[Test build #2076 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2076/consoleFull)**
 for PR 9768 at commit 
[`82a261d`](https://github.com/apache/spark/commit/82a261d11fdf86cf15bc19fc56f796725973fa43).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11775][PYSPARK][SQL] Allow PySpark to r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9766#issuecomment-157350327
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11775][PYSPARK][SQL] Allow PySpark to r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9766#issuecomment-157350194
  
**[Test build #46078 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46078/consoleFull)**
 for PR 9766 at commit 
[`dd1e269`](https://github.com/apache/spark/commit/dd1e269be3c9d976428297da532d57e24b55af55).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * `
throw new IOException(s\"UDF class $`\n  * `throw new 
IOException(s\"It is invalid to implement multiple UDF interfaces, UDF class 
$`\n  * `case n => logError(s\"UDF class with $`\n  * `
logError(s\"Can not instantiate class $`\n  * `  case e: 
ClassNotFoundException => logError(s\"Can not load class $`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11750][SQL] revert SPARK-11727 and code...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/9726#issuecomment-157363022
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11740][Streaming]Fix the race condition...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/9707#issuecomment-157334797
  
LGTM.
I wonder whether we can add a unit test to test this behavior. We could 
create a CheckpointWriter with a mock JobGenerator, and then test whether the 
latest checkpoint time is respected. 

BTW, I would like to merge this to older branches like 1.5 and 1.4 as well. 

@harishreedharan Can you also take a look at this? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11684] [R] [ML] [Doc] Update SparkR glm...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9727#issuecomment-157334397
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11016] Move RoaringBitmap to explicit K...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9748#issuecomment-157340460
  
Lacking background knowledge to sign off this one. Referenced discussions 
in #9243, https://github.com/metamx/spark/pull/1, SPARK-5949, and SPARK-11016. 
If understand it correctly, currently `RoaringBitmap` isn't a direct dependency 
of Spark, instead it's just a transitive dependency, and Kryo needs it to work 
properly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11775][PYSPARK][SQL] Allow PySpark to r...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9766#issuecomment-157350329
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46078/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7888#issuecomment-157335125
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46073/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157338196
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46080/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157338193
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11016] Move RoaringBitmap to explicit K...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/9748#discussion_r45046892
  
--- Diff: 
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -377,6 +374,55 @@ private[serializer] object KryoSerializer {
 classOf[BoundedPriorityQueue[_]],
 classOf[SparkConf]
   )
+
+  private val toRegisterSerializer = Map[Class[_], KryoClassSerializer[_]](
+classOf[RoaringBitmap] -> new KryoClassSerializer[RoaringBitmap](){
+  override def write(kryo: Kryo, output: KryoOutput, bitmap: 
RoaringBitmap): Unit =
+   bitmap.serialize(new KryoOutputDataOutputBridge(output))
+  override def read(kryo: Kryo, input: KryoInput, clazz: 
Class[RoaringBitmap]):
+   RoaringBitmap = {
--- End diff --

Indentation is off. And needn't to break the line here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11016] Move RoaringBitmap to explicit K...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/9748#discussion_r45046897
  
--- Diff: 
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -377,6 +374,55 @@ private[serializer] object KryoSerializer {
 classOf[BoundedPriorityQueue[_]],
 classOf[SparkConf]
   )
+
+  private val toRegisterSerializer = Map[Class[_], KryoClassSerializer[_]](
+classOf[RoaringBitmap] -> new KryoClassSerializer[RoaringBitmap](){
+  override def write(kryo: Kryo, output: KryoOutput, bitmap: 
RoaringBitmap): Unit =
+   bitmap.serialize(new KryoOutputDataOutputBridge(output))
+  override def read(kryo: Kryo, input: KryoInput, clazz: 
Class[RoaringBitmap]):
+   RoaringBitmap = {
+val ret = new RoaringBitmap
+ret.deserialize(new KryoInputDataInputBridge(input))
+ret
+  }
+}
+  )
+}
+
+private[serializer] class KryoInputDataInputBridge(input : KryoInput) 
extends DataInput {
--- End diff --

Remove the space before `:`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11016] Move RoaringBitmap to explicit K...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9748#issuecomment-157341226
  
Yep, LGTM, thanks for going back and taking care of this loose end @davies 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11694] [SQL] Backports #9754

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9763#issuecomment-157341345
  
LGTM pending Jenkins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11694] [SQL] Backports #9754

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9763#issuecomment-157342417
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46075/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11694] [SQL] Backports #9754

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9763#issuecomment-157342415
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11767] [SQL] limit the size of caced ba...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9760#issuecomment-157342723
  
This changes LGTM, but I don't quite understand this line in the PR 
description:

> ... This PR introduce a hard limit as 4M for total columns (up to 50 
columns of uncompressed primitive columns).

Where is the "50" from? Is it a hard limit defined somewhere else or an 
estimation of average use case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11750][SQL] revert SPARK-11727 and code...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9726#issuecomment-157354774
  
**[Test build #46070 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46070/consoleFull)**
 for PR 9726 at commit 
[`37fa83f`](https://github.com/apache/spark/commit/37fa83f4e359c62a69b3ecfd17922087e8f79a96).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11740][Streaming]Fix the race condition...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/9707#discussion_r45044684
  
--- Diff: python/pyspark/streaming/tests.py ---
@@ -699,11 +698,11 @@ def check_output(n):
 # Verify that getOrCreate() uses existing SparkContext
 self.ssc.stop(True, True)
 time.sleep(1)
-sc = SparkContext(SparkConf())
--- End diff --

is this change needed for this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9264#issuecomment-157338424
  
**[Test build #2075 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2075/consoleFull)**
 for PR 9264 at commit 
[`9da3533`](https://github.com/apache/spark/commit/9da3533e6887a809a0c92010dcfdb2ca7df3dce8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11694] [SQL] Backports #9754

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9763#issuecomment-157342265
  
**[Test build #46075 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46075/consoleFull)**
 for PR 9763 at commit 
[`1f252ec`](https://github.com/apache/spark/commit/1f252ecf3527b8566c8f5c7b87a1a2af9aa33d06).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10946][SQL]JDBC - Use Statement.execute...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9733#discussion_r45052370
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
 ---
@@ -120,7 +120,7 @@ private[sql] object JDBCRDD extends Logging {
 val dialect = JdbcDialects.get(url)
 val conn: Connection = getConnector(properties.getProperty("driver"), 
url, properties)()
 try {
-  val rs = conn.prepareStatement(s"SELECT * FROM $table WHERE 
1=0").executeQuery()
+  val rs = conn.createStatement.executeQuery(s"SELECT * FROM $table 
WHERE 1=0")
--- End diff --

Ah hang on a sec, I see you've changed some non-DDL statements in this 
version of the PR. The change is just about avoiding `PreparedStatement` for 
DDL right? My earlier comments were about closing resources for a bunch of 
queries including non-DDL. But if you're only addressing DDL, then your last PR 
was fine (before it was corrupted).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[jbonofre]

Github user jbonofre commented on a diff in the pull request:

https://github.com/apache/spark/pull/9165#discussion_r45055081
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/sorttable.js 
---
@@ -169,7 +169,7 @@ sorttable = {
 for (var i=0; i

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-157363885
  
**[Test build #2072 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2072/consoleFull)**
 for PR 9736 at commit 
[`253235a`](https://github.com/apache/spark/commit/253235aba449d0609824f6ab743e95758b104e19).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157367136
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46084/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157367132
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11692][SQL] Support for Parquet logical...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/9658#issuecomment-157373552
  
Oh. I just got confused for a bit. Sorry.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11771][YARN][TRIVIAL] maximum memory in...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9758#discussion_r45045562
  
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -258,7 +258,8 @@ private[spark] class Client(
 if (executorMem > maxMem) {
   throw new IllegalArgumentException(s"Required executor memory 
(${args.executorMemory}" +
 s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem 
MB) of this cluster! " +
-"Please increase the value of 
'yarn.scheduler.maximum-allocation-mb'.")
+"Please increase the value of 
'yarn.scheduler.maximum-allocation-mb' and" +
--- End diff --

"and/or"? it could really be due to either one -- the max the scheduler 
will schedule, or the limit on the max size of a container. It may not need an 
increase in both.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11779][DOCS] Fix reference to deprecate...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9768#issuecomment-157339909
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11779][DOCS] Fix reference to deprecate...

[philipphoffmann]

GitHub user philipphoffmann opened a pull request:

https://github.com/apache/spark/pull/9768

[SPARK-11779][DOCS] Fix reference to deprecated MESOS_NATIVE_LIBRARY

MESOS_NATIVE_LIBRARY was renamed in favor of MESOS_NATIVE_JAVA_LIBRARY. 
This commit fixes the reference in the documentation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/philipphoffmann/spark patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9768.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9768


commit 82a261d11fdf86cf15bc19fc56f796725973fa43
Author: Philipp Hoffmann 
Date:   2015-11-17T11:10:56Z

[SPARK-11779][DOCS] Fix reference to deprecated MESOS_NATIVE_LIBRARY

MESOS_NATIVE_LIBRARY was renamed in favor of MESOS_NATIVE_JAVA_LIBRARY. 
This commit fixes the reference in the documentation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11692][SQL] Support for Parquet logical...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/9658#issuecomment-157341844
  
Please note that https://github.com/apache/spark/pull/9754 updated 
unintentionally this to clean up at mater branch however, that is supposed to 
be merged with branch 1.6 and for this version 1.7.
I will give a alert to you or make a PR to backport this as soon as branch 
1.7 is available. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11779][DOCS] Fix reference to deprecate...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9768#issuecomment-157341519
  
That's correct as of the version supported in Spark 1.5?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11692][SQL] Support for Parquet logical...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9658#issuecomment-157343609
  
Hm, I don't quite get it... So this PR is only for master (targeting 1.7). 
I don't think we need to backport this one to anywhere else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3580] Add Consistent Method To Get Numb...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9767#issuecomment-157353135
  
Yes, ideally. @JoshRosen am I right that we have to add the new method to 
the `JavaRDDLike` trait or the abstract class that implements it in order to 
avoid binary incompatibility? Both?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11773][SPARKR] Implement collection fun...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9764#issuecomment-157362335
  
**[Test build #46079 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46079/consoleFull)**
 for PR 9764 at commit 
[`ece78e8`](https://github.com/apache/spark/commit/ece78e83fe4328aab3574f8451f6766ae74bc522).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157367766
  
**[Test build #2074 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2074/consoleFull)**
 for PR 9165 at commit 
[`9a79862`](https://github.com/apache/spark/commit/9a79862be3e7aa8ec4cf6e57e0e826b211f81e27).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157367868
  
**[Test build #2078 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2078/consoleFull)**
 for PR 9165 at commit 
[`5526031`](https://github.com/apache/spark/commit/5526031a29fab812329ef35c0f25e6d12c2f4fd5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11195][CORE] Use correct classloader fo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9367#issuecomment-157369352
  
  [Test build #2073 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2073/console)
 for   PR 9367 at commit 
[`fadf2ca`](https://github.com/apache/spark/commit/fadf2ca57e21d5f2600d4ec3e504e20db596c35c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11694] [SQL] Backports #9754

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9763#issuecomment-157371768
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46082/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11694] [SQL] Backports #9754

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9763#issuecomment-157371764
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8658] [SQL] [FOLLOW-UP] AttributeRefere...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/9761#issuecomment-157414057
  
@nongli I saw you have a related discussion with @chenghao-intel . The 
failed test case was introduced in your PR 
https://github.com/apache/spark/pull/9480. I am not sure the root reason why we 
intentionally exclude `name` from `hashCode`, but the original `equals` include 
`name`. It breaks a general principle of hashCode function design:

```
An object’s hashCode method must take the same fields into account as its 
equals method. 
``` 

Based on my understanding, in a case-sensitive HiveContext, we still should 
detect their differences when the case of `name` is different but the `exprId` 
values are the same 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7753#issuecomment-157418537
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46091/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7753#issuecomment-157418522
  
**[Test build #46091 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46091/consoleFull)**
 for PR 7753 at commit 
[`75e63c3`](https://github.com/apache/spark/commit/75e63c366fcee3fed5cfe5c1066ab9727fd1f574).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`class ExecutorMetrics extends Serializable `\n  * `class TransportMetrics (`\n 
 * `class MemoryListener extends SparkListener `\n  * `class MemoryUIInfo `\n  
* `class TransportMemSize `\n  * `case class MemTime(memorySize: Long = 
System.currentTimeMillis, timeStamp: Long = 0L)`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11195][CORE] Use correct classloader fo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9367#issuecomment-157421084
  
  [Test build #2079 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2079/console)
 for   PR 9367 at commit 
[`fadf2ca`](https://github.com/apache/spark/commit/fadf2ca57e21d5f2600d4ec3e504e20db596c35c).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11057] [SQL] Add correlation and covari...

[NarineK]

Github user NarineK commented on the pull request:

https://github.com/apache/spark/pull/9366#issuecomment-157420688
  
can someone from Spark SQL committers or experts also look at this ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11195][CORE] Use correct classloader fo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9367#issuecomment-157422891
  
  [Test build #2080 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2080/console)
 for   PR 9367 at commit 
[`fadf2ca`](https://github.com/apache/spark/commit/fadf2ca57e21d5f2600d4ec3e504e20db596c35c).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[jbonofre]

Github user jbonofre commented on a diff in the pull request:

https://github.com/apache/spark/pull/9165#discussion_r45083392
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/sorttable.js 
---
@@ -169,7 +169,7 @@ sorttable = {
 for (var i=0; i

[GitHub] spark pull request: [SPARK-11281][SPARKR] Add tests covering the i...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9743#issuecomment-157415429
  
**[Test build #46089 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46089/consoleFull)**
 for PR 9743 at commit 
[`0508988`](https://github.com/apache/spark/commit/0508988d537a74f33df1f6e22adc2bf98ad767a1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/7753#issuecomment-157418529
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11057] [SQL] Add correlation and covari...

[NarineK]

Github user NarineK commented on the pull request:

https://github.com/apache/spark/pull/9366#issuecomment-157419321
  
yes, there is even a test case which covers that case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10946][SQL]JDBC - Use Statement.execute...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9733#issuecomment-157419417
  
You don't need a new PR, just push a new commit here. It's better, as it 
preserves the history of the discussion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: rmse was wrongly calculated

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9771#issuecomment-157421443
  
LGTM. This is on the border of needing a JIRA IMHO.
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11191] [SQL] [FOLLOW-UP] Cleans up unne...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9737#issuecomment-157421259
  
Also backported to branch-1.6.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9165#discussion_r45082916
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/sorttable.js 
---
@@ -169,7 +169,7 @@ sorttable = {
 for (var i=0; i

[GitHub] spark pull request: [SPARK-11725][SQL] correctly handle null input...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9770#issuecomment-157410913
  
**[Test build #46090 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46090/consoleFull)**
 for PR 9770 at commit 
[`a8a3067`](https://github.com/apache/spark/commit/a8a30674ce531c9cd10107200a3f72f9539cd8f6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11281][SPARKR] Add tests covering the i...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9743#issuecomment-157415572
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11281][SPARKR] Add tests covering the i...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9743#issuecomment-157415577
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46089/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/7753#issuecomment-157417841
  
**[Test build #46091 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46091/consoleFull)**
 for PR 7753 at commit 
[`75e63c3`](https://github.com/apache/spark/commit/75e63c366fcee3fed5cfe5c1066ab9727fd1f574).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8658] [SQL] [FOLLOW-UP] AttributeRefere...

[nongli]

Github user nongli commented on the pull request:

https://github.com/apache/spark/pull/9761#issuecomment-157421827
  
This change looks good to me. The failing test is not valid anymore and 
should be changed to verify that the hash codes are not equal.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: rmse was wrongly calculated

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9771#issuecomment-157421857
  
**[Test build #2081 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2081/consoleFull)**
 for PR 9771 at commit 
[`4cee53c`](https://github.com/apache/spark/commit/4cee53c18ed0c1c3e77c1101054044adf3250caa).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157336274
  
**[Test build #46081 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46081/consoleFull)**
 for PR 9740 at commit 
[`8b70477`](https://github.com/apache/spark/commit/8b70477048208b41ba03b524abc510191633e5e0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11530] [MLLIB] Return eigenvalues with ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9736#issuecomment-157336366
  
**[Test build #2072 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2072/consoleFull)**
 for PR 9736 at commit 
[`253235a`](https://github.com/apache/spark/commit/253235aba449d0609824f6ab743e95758b104e19).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11732] Removes some MiMa false positive...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9697#issuecomment-157336315
  
LGTM @pwendell as I can't see why MiMa would need to look at private 
classes of any type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11779][DOCS] Fix reference to deprecate...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9768#issuecomment-157342003
  
**[Test build #2076 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2076/consoleFull)**
 for PR 9768 at commit 
[`82a261d`](https://github.com/apache/spark/commit/82a261d11fdf86cf15bc19fc56f796725973fa43).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11750][SQL] revert SPARK-11727 and code...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9726#issuecomment-157354977
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46070/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11750][SQL] revert SPARK-11727 and code...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9726#issuecomment-157354973
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11750][SQL] revert SPARK-11727 and code...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9726#issuecomment-157364779
  
**[Test build #46083 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46083/consoleFull)**
 for PR 9726 at commit 
[`37fa83f`](https://github.com/apache/spark/commit/37fa83f4e359c62a69b3ecfd17922087e8f79a96).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7012][SQL] Add support for NOT NULL mod...

[sabhyankar]

Github user sabhyankar commented on a diff in the pull request:

https://github.com/apache/spark/pull/8746#discussion_r45070100
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/sources/DDLTestSuite.scala ---
@@ -113,4 +113,23 @@ class DDLTestSuite extends DataSourceTest with 
SharedSQLContext {
 assert(attributes.map(_.name) === Seq("col_name", "data_type", 
"comment"))
 assert(attributes.map(_.dataType).toSet === Set(StringType))
   }
+
+  test("SPARK-7012 Create table statement should support NOT NULL modifier 
for columns") {
+withTempPath { dir =>
+  val path = dir.getCanonicalPath
+  sql(
+  s"""
+ |CREATE TEMPORARY TABLE tempTableDDL
+ |( tCol1 INT NOT NULL,
+ |  tCol2 STRING
+ |)
+ |USING parquet
+ |OPTIONS (
+ |  path '$path'
+ |)
+   """.stripMargin
+  )
+  caseInsensitiveContext.dropTempTable("tempTableDDL")
--- End diff --

Hi @smola @cloud-fan - I will take a look at the test case this week and 
update you guys


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157398049
  
**[Test build #46085 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46085/consoleFull)**
 for PR 9165 at commit 
[`5526031`](https://github.com/apache/spark/commit/5526031a29fab812329ef35c0f25e6d12c2f4fd5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11755] [R] SparkR should export "predic...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9732#issuecomment-157397984
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157398356
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157398361
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46085/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11725][SQL] correctly handle null input...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9770#issuecomment-157401188
  
**[Test build #46088 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46088/consoleFull)**
 for PR 9770 at commit 
[`e258b6c`](https://github.com/apache/spark/commit/e258b6ce46bfa0576730d5c46728a4fc8fc7bc24).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: rmse was wrongly calculated

[vivkul]

GitHub user vivkul opened a pull request:

https://github.com/apache/spark/pull/9771

rmse was wrongly calculated

It was multiplying with U instaed of dividing by U

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vivkul/spark patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9771.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9771


commit 4cee53c18ed0c1c3e77c1101054044adf3250caa
Author: Viveka Kulharia 
Date:   2015-11-17T15:17:12Z

rmse was wrongly calculated

It was multiplying with U instaed of dividing by U




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11750][SQL] revert SPARK-11727 and code...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9726#issuecomment-157399730
  
**[Test build #46083 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46083/consoleFull)**
 for PR 9726 at commit 
[`37fa83f`](https://github.com/apache/spark/commit/37fa83f4e359c62a69b3ecfd17922087e8f79a96).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: rmse was wrongly calculated

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9771#issuecomment-157403077
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11281][SPARKR] Add tests covering the i...

[zero323]

Github user zero323 commented on a diff in the pull request:

https://github.com/apache/spark/pull/9743#discussion_r45074174
  
--- Diff: R/pkg/inst/tests/test_sparkSQL.R ---
@@ -294,9 +294,21 @@ test_that("create DataFrame with complex types", {
 test_that("create DataFrame from a data.frame with complex types", {
   ldf <- data.frame(row.names=1:2)
   ldf$a_list <- list(list(1, 2), list(3, 4))
+  ldf$an_envir <- c(as.environment(list(a=1, b=2)), 
as.environment(list(c=3)))
+
   sdf <- createDataFrame(sqlContext, ldf)
+  collected <- collect(sdf)
+
+  expect_equivalent(ldf[, 1, FALSE], collected[, 1, FALSE])
+
+  asSortedList <- function(e) {
+  tmp <- as.list(e)
+  tmp[sort(names(tmp))]
+  }
 
-  expect_equivalent(ldf, collect(sdf))
+  expect_equal(
--- End diff --

@sun-rui You're right. Both changes should work just fine.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11281][SPARKR] Add tests covering the i...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9743#issuecomment-157409861
  
**[Test build #46089 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46089/consoleFull)**
 for PR 9743 at commit 
[`0508988`](https://github.com/apache/spark/commit/0508988d537a74f33df1f6e22adc2bf98ad767a1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11447][SQL] change NullType to StringTy...

[kevinyu98]

Github user kevinyu98 commented on a diff in the pull request:

https://github.com/apache/spark/pull/9720#discussion_r45069001
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
 ---
@@ -280,6 +280,12 @@ object HiveTypeCoercion {
   case p @ BinaryComparison(left @ DateType(), right @ 
TimestampType()) =>
 p.makeCopy(Array(Cast(left, StringType), Cast(right, StringType)))
 
+  // Checking NullType
+  case p @ BinaryComparison(left @ StringType(), right @ NullType()) =>
+p.makeCopy(Array(left, Literal.create(null, StringType)))
+  case p @ BinaryComparison(left @ NullType(), right @ StringType()) =>
+p.makeCopy(Array(Literal.create(null, StringType), right))
+
   case p @ BinaryComparison(left @ StringType(), right) if 
right.dataType != StringType =>
 p.makeCopy(Array(Cast(left, DoubleType), right))
--- End diff --

@yhuai @cloud-fan : sure, I will not do that. I will try to run more 
testing to see if anything is broken.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-157399487
  
**[Test build #2078 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2078/consoleFull)**
 for PR 9165 at commit 
[`5526031`](https://github.com/apache/spark/commit/5526031a29fab812329ef35c0f25e6d12c2f4fd5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11755] [R] SparkR should export "predic...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9732#issuecomment-157397991
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46087/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11750][SQL] revert SPARK-11727 and code...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9726#issuecomment-157400218
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org