[GitHub] spark pull request: [SPARK-12361][PYSPARK][TESTS] Should set PYSPA...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/10322#issuecomment-165221194
  
LGTM, so I'm going to merge this into master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9690][ML][PYTHON] pyspark CrossValidato...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10268#issuecomment-165221379
  
**[Test build #47844 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47844/consoleFull)**
 for PR 10268 at commit 
[`9bf75ad`](https://github.com/apache/spark/commit/9bf75adc5f39cfa7015b6c9934350bfa40470010).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`class CrossValidator(Estimator, HasSeed):`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12361][PYSPARK][TESTS] Should set PYSPA...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10322


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165224554
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11749][Streaming] Duplicate creating th...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9765#discussion_r47823645
  
--- Diff: 
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -715,6 +736,35 @@ class CheckpointSuite extends TestSuiteBase with 
DStreamCheckpointTester {
 }
   }
 
+  test("DStreamCheckpointData.restore invoking times") {
+var clock: ManualClock = null
--- End diff --

nit: remove this unused line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11749][Streaming] Duplicate creating th...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9765#discussion_r47823739
  
--- Diff: 
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -715,6 +736,35 @@ class CheckpointSuite extends TestSuiteBase with 
DStreamCheckpointTester {
 }
   }
 
+  test("DStreamCheckpointData.restore invoking times") {
+var clock: ManualClock = null
+val outputBuffer = new ArrayBuffer[Seq[Seq[Int]]] with 
SynchronizedBuffer[Seq[Seq[Int]]]
--- End diff --

You can remove `outputBuffer` and change `val outputStream = new 
TestOutputStreamWithPartitions(mappedDStream, outputBuffer)` to `val 
outputStream = new TestOutputStreamWithPartitions(mappedDStream)` since the 
output buffer is not necessary now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12220][Core]Make Utils.fetchFile suppor...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10208#issuecomment-165226014
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165226009
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11100][SQL] HiveThriftServer not regist...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9113#issuecomment-165225983
  
**[Test build #47849 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47849/consoleFull)**
 for PR 9113 at commit 
[`43769ea`](https://github.com/apache/spark/commit/43769ea659a14919fb432fdd423387a57791c744).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165226005
  
**[Test build #47845 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47845/consoleFull)**
 for PR 10329 at commit 
[`f75815b`](https://github.com/apache/spark/commit/f75815b9c4669b2871029cb35f8c95960d6fd54f).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165225775
  
@skyluc looks like there is a style violation with this patch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345][MESOS] Filter SPARK_HOME when su...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10332#issuecomment-165227460
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11608][MLLIB][DOC] Added migration guid...

[jkbradley]

Github user jkbradley commented on the pull request:

https://github.com/apache/spark/pull/10235#issuecomment-165227363
  
OK thanks!
Merging with master and branch-1.6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345][MESOS] Filter SPARK_HOME when su...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10332#issuecomment-165227299
  
**[Test build #47830 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47830/consoleFull)**
 for PR 10332 at commit 
[`baea28f`](https://github.com/apache/spark/commit/baea28f54406a58ae313d1a8428d985e70b3116a).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11100][SQL] HiveThriftServer not regist...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9113#issuecomment-165231454
  
**[Test build #47849 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47849/consoleFull)**
 for PR 9113 at commit 
[`43769ea`](https://github.com/apache/spark/commit/43769ea659a14919fb432fdd423387a57791c744).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11100][SQL] HiveThriftServer not regist...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9113#issuecomment-165231601
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47849/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165233024
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/10335#issuecomment-165238477
  
Sure, will do It! Thank you for your guidance!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-165241236
  
**[Test build #47838 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47838/consoleFull)**
 for PR 8880 at commit 
[`fd2664b`](https://github.com/apache/spark/commit/fd2664b08d443d7c97cab99246146d25698ce5fb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11097][Core]Add connectionEstablished c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10301#issuecomment-165244297
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47834/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11097][Core]Add connectionEstablished c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10301#issuecomment-165244293
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12350] [core] Don't log errors when req...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10337#issuecomment-165246287
  
**[Test build #47854 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47854/consoleFull)**
 for PR 10337 at commit 
[`b99a470`](https://github.com/apache/spark/commit/b99a47086826bd1fca64e6dec839ce372bfa024e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11749][Streaming] Duplicate creating th...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9765#issuecomment-165220595
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47832/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165223713
  
**[Test build #2221 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2221/consoleFull)**
 for PR 10329 at commit 
[`f75815b`](https://github.com/apache/spark/commit/f75815b9c4669b2871029cb35f8c95960d6fd54f).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11904] [PySpark] reduceByKeyAndWindow d...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9888#issuecomment-165225716
  
**[Test build #47848 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47848/consoleFull)**
 for PR 9888 at commit 
[`6730f72`](https://github.com/apache/spark/commit/6730f72d2d9aa2c535abc9719e589369cc7b4cdb).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11904] [PySpark] reduceByKeyAndWindow d...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9888#issuecomment-165225719
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165222829
  
**[Test build #47845 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47845/consoleFull)**
 for PR 10329 at commit 
[`f75815b`](https://github.com/apache/spark/commit/f75815b9c4669b2871029cb35f8c95960d6fd54f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11904] [PySpark] reduceByKeyAndWindow d...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9888#issuecomment-165225722
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47848/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] Add missing interpolation in NettyRPCE...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/10334#issuecomment-165226512
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165230905
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10335#issuecomment-165232112
  
**[Test build #47852 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47852/consoleFull)**
 for PR 10335 at commit 
[`2aab4d6`](https://github.com/apache/spark/commit/2aab4d648fa634ec427e37ae82b0328fad159720).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47829608
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -126,6 +127,69 @@ case class Sample(
   }
 }
 
+case class Range(
+start: Long,
+step: Long,
+numSlices: Int,
+numElements: BigInt,
+output: Seq[Attribute])
+  extends LeafNode
+{
--- End diff --

Nit: this should be on the previous line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47830240
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -210,6 +210,37 @@ case class Sort(
   override def output: Seq[Attribute] = child.output
 }
 
+case class Range(
+start: Long,
+end: Long,
+step: Long,
+numSlices: Int) extends LeafNode {
+  require(step != 0, "step cannot be 0")
+  val numElements: BigInt = {
+val safeStart = BigInt(start)
+val safeEnd = BigInt(end)
+if ((safeEnd - safeStart) % step == 0 || safeEnd > safeStart ^ step > 
0) {
+  (safeEnd - safeStart) / step
+} else {
+  // the remainder has the same sign with range, could add 1 more
+  (safeEnd - safeStart) / step + 1
+}
+  }
+
+  val output: Seq[Attribute] =
--- End diff --

I think it would be better to make this an argument to the constructor and 
have a factory in the companion object that creates one automatically.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11097][Core]Add connectionEstablished c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10301#issuecomment-165244012
  
**[Test build #47834 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47834/consoleFull)**
 for PR 10301 at commit 
[`b36d776`](https://github.com/apache/spark/commit/b36d776f648e78b21384b7e17dd118babd5bf833).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12350] [core] Don't log errors when req...

[vanzin]

GitHub user vanzin opened a pull request:

https://github.com/apache/spark/pull/10337

[SPARK-12350] [core] Don't log errors when requested stream is not found.

If a client requests a non-existent stream, just send a failure message
back, without logging any error on the server side (since it's not a
server error).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vanzin/spark SPARK-12350

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10337.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10337


commit b99a47086826bd1fca64e6dec839ce372bfa024e
Author: Marcelo Vanzin 
Date:   2015-12-16T21:01:41Z

[SPARK-12350] [core] Don't log errors when requested stream is not found.

If a client requests a non-existent stream, just send a failure message
back, without logging any error on the server side (since it's not a
server error).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47834244
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -210,6 +210,37 @@ case class Sort(
   override def output: Seq[Attribute] = child.output
 }
 
+case class Range(
+start: Long,
+end: Long,
+step: Long,
+numSlices: Int) extends LeafNode {
+  require(step != 0, "step cannot be 0")
+  val numElements: BigInt = {
+val safeStart = BigInt(start)
+val safeEnd = BigInt(end)
+if ((safeEnd - safeStart) % step == 0 || safeEnd > safeStart ^ step > 
0) {
--- End diff --

I don't think this is the place where we should start speeding thing up 
:)... I'd rather have something which is easier to read.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] Add missing interpolation in NettyRPCE...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10334#issuecomment-165247391
  
**[Test build #47842 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47842/consoleFull)**
 for PR 10334 at commit 
[`a853e63`](https://github.com/apache/spark/commit/a853e63250e57e4420afff547a962f20259af652).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12321][SQL] JSON format for TreeNode (u...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10311#issuecomment-165175811
  
**[Test build #47819 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47819/consoleFull)**
 for PR 10311 at commit 
[`6b293ba`](https://github.com/apache/spark/commit/6b293ba5d8c806c914437895a22ce85d86a4418a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10775][UI]add search keywords in histor...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8885#issuecomment-165184442
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345][MESOS] Filter SPARK_HOME when su...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10332#issuecomment-165194859
  
**[Test build #47830 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47830/consoleFull)**
 for PR 10332 at commit 
[`baea28f`](https://github.com/apache/spark/commit/baea28f54406a58ae313d1a8428d985e70b3116a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165202252
  
I will go ahead and do a quick test on standalone cluster mode with this 
fix and report back.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165202088
  
This seems like a better place to filter it out. My only concern is that 
this one also affects standalone cluster mode, whereas #10332 is totally a 
Mesos thing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345][MESOS] Filter SPARK_HOME when su...

[skyluc]

Github user skyluc commented on the pull request:

https://github.com/apache/spark/pull/10332#issuecomment-165204770
  
Code LGTM. Unfortunately, I cannot try it before a couple of hours.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165204839
  
cc @liancheng 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12318][SPARKR] Save mode in SparkR shou...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10290#issuecomment-165204897
  
LGTM. Merging into master (and not `branch-1.6` as we probably don't want 
to change the API in a minor release like 1.6.1). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345][MESOS] Filter SPARK_HOME when su...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/10332#issuecomment-165210134
  
LGTM merging into master and 1.6. Just FYI I might revert this patch in 
master because I believe #10329 is a better fix in the long run, but for now 
let's just unblock the release.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345][MESOS] Filter SPARK_HOME when su...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/10332#discussion_r47816823
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala ---
@@ -94,7 +94,12 @@ private[mesos] class MesosSubmitRequestServlet(
 val driverMemory = sparkProperties.get("spark.driver.memory")
 val driverCores = sparkProperties.get("spark.driver.cores")
 val appArgs = request.appArgs
-val environmentVariables = request.environmentVariables
+// We don't want to pass down SPARK_HOME when launching Spark apps
+// with Mesos cluster mode since it's populated by default on the 
client and it will
+// cause spark-submit script to look for files in SPARK_HOME instead.
+// We only need the ability to specify where to find spark-submit 
script
+// which user can user spark.executor.home or spark.home 
configurations.
--- End diff --

I would add (SPARK-12345) here, but I'll fix this myself on merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12321][SQL] JSON format for TreeNode (u...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10311#issuecomment-165209891
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12048][SQL] Part 2 Prevent to close JDB...

[tedyu]

Github user tedyu commented on the pull request:

https://github.com/apache/spark/pull/10320#issuecomment-165209976
  
I compared JDBCRDD.scala with JdbcRDD.scala
From what I can tell according to the usage of java.sql.Connection and 
java.sql.ResultSet, the proposed change is needed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6518][MLlib][Example][DOC] Add example ...

[jkbradley]

Github user jkbradley commented on the pull request:

https://github.com/apache/spark/pull/9952#issuecomment-165210239
  
LGTM, merging with master and branch-1.6
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9690][ML][PYTHON] pyspark CrossValidato...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10268#issuecomment-165221624
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9690][ML][PYTHON] pyspark CrossValidato...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10268#issuecomment-165221626
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47844/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4621][shuffle]Shuffle index can be cach...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10277#issuecomment-165221559
  
**[Test build #47827 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47827/consoleFull)**
 for PR 10277 at commit 
[`3485e0f`](https://github.com/apache/spark/commit/3485e0f6680732bd055789d68f15fc3189ab54ad).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`public class ShuffleIndexCache `\n  * `public class ShuffleIndexRecord `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10618] [Mesos] Refactoring scheduling c...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/10326#issuecomment-165224832
  
ok to test (@dragos you can also say this by the way)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11749][Streaming] Duplicate creating th...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9765#issuecomment-165224917
  
Just two nits. Otherwise LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10325#issuecomment-165229350
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10325#issuecomment-165229169
  
**[Test build #47831 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47831/consoleFull)**
 for PR 10325 at commit 
[`87e2d0d`](https://github.com/apache/spark/commit/87e2d0d2526b9d87cdd9fb041a7a6f0b5c731039).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165232552
  
**[Test build # has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder//consoleFull)**
 for PR 10333 at commit 
[`e418358`](https://github.com/apache/spark/commit/e41835804b818724f8c28c12e9606b4d4052fe37).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165232881
  
**[Test build #47835 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47835/consoleFull)**
 for PR 10333 at commit 
[`fd4c945`](https://github.com/apache/spark/commit/fd4c9458fd57bcebe5b8fd09e101c7a937131454).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165233027
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47835/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11171][SPARK-11237][SPARK-11241][ML] Tr...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/9207#issuecomment-165233108
  
ping @jkbradley 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47828090
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -126,6 +127,69 @@ case class Sample(
   }
 }
 
+case class Range(
+start: Long,
+step: Long,
+numSlices: Int,
+numElements: BigInt,
+output: Seq[Attribute])
+  extends LeafNode
+{
+  override def outputsUnsafeRows: Boolean = true
+
+  protected override def doExecute(): RDD[InternalRow] = {
+sqlContext
+  .sparkContext
+  .parallelize(0 until numSlices, numSlices)
+  .mapPartitionsWithIndex((i, _) => {
+val partitionStart = (i * numElements) / numSlices * step + start
+val partitionEnd = (((i + 1) * numElements) / numSlices) * step + 
start
+def getSafeMargin(bi: BigInt): Long =
+  if (bi.isValidLong) {
+bi.toLong
+  } else if (bi > 0) {
+Long.MaxValue
+  } else {
+Long.MinValue
+  }
+val safePartitionStart = getSafeMargin(partitionStart)
+val safePartitionEnd = getSafeMargin(partitionEnd)
+val bufferHolder = new BufferHolder(LongType.defaultSize)
+val unsafeRow = new UnsafeRow
+
+new Iterator[InternalRow] {
+  private[this] var number: Long = safePartitionStart
+  private[this] var overflow: Boolean = false
+
+  override def hasNext =
+if (!overflow) {
--- End diff --

Is it even possible to overflow? We know the ```size```, ```begin``` and 
the ```end``` values when we create the object; an overflow should become there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12186] [WEB UI] Send the complete reque...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10180#issuecomment-165233273
  
**[Test build #47839 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47839/consoleFull)**
 for PR 10180 at commit 
[`780ee62`](https://github.com/apache/spark/commit/780ee624b23703b11c4c90b74b80af8202abd21e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47828506
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -210,6 +210,37 @@ case class Sort(
   override def output: Seq[Attribute] = child.output
 }
 
+case class Range(
+start: Long,
+end: Long,
+step: Long,
+numSlices: Int) extends LeafNode {
+  require(step != 0, "step cannot be 0")
+  val numElements: BigInt = {
+val safeStart = BigInt(start)
+val safeEnd = BigInt(end)
+if ((safeEnd - safeStart) % step == 0 || safeEnd > safeStart ^ step > 
0) {
--- End diff --

```xor```s are typically really hard to follow. I think this might be 
easier: ```safeEnd > safeStart == step > 0```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12186] [WEB UI] Send the complete reque...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/10180#issuecomment-165240392
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165230907
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47851/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47827840
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -126,6 +127,69 @@ case class Sample(
   }
 }
 
+case class Range(
+start: Long,
+step: Long,
+numSlices: Int,
+numElements: BigInt,
+output: Seq[Attribute])
+  extends LeafNode
+{
+  override def outputsUnsafeRows: Boolean = true
+
+  protected override def doExecute(): RDD[InternalRow] = {
+sqlContext
+  .sparkContext
+  .parallelize(0 until numSlices, numSlices)
+  .mapPartitionsWithIndex((i, _) => {
+val partitionStart = (i * numElements) / numSlices * step + start
+val partitionEnd = (((i + 1) * numElements) / numSlices) * step + 
start
+def getSafeMargin(bi: BigInt): Long =
+  if (bi.isValidLong) {
+bi.toLong
+  } else if (bi > 0) {
+Long.MaxValue
+  } else {
+Long.MinValue
+  }
+val safePartitionStart = getSafeMargin(partitionStart)
+val safePartitionEnd = getSafeMargin(partitionEnd)
+val bufferHolder = new BufferHolder(LongType.defaultSize)
+val unsafeRow = new UnsafeRow
+
+new Iterator[InternalRow] {
+  private[this] var number: Long = safePartitionStart
+  private[this] var overflow: Boolean = false
+
+  override def hasNext =
+if (!overflow) {
+  if (step > 0) {
+number < safePartitionEnd
+  } else {
+number > safePartitionEnd
+  }
+} else false
+
+  override def next() = {
+val ret = number
+number += step
+if (number < ret ^ step < 0) {
+  // we have Long.MaxValue + Long.MaxValue < Long.MaxValue
+  // and Long.MinValue + Long.MinValue > Long.MinValue, so iff 
the step causes a step
+  // back, we are pretty sure that we have an overflow.
+  overflow = true
+}
+
+bufferHolder.reset()
+unsafeRow.pointTo(bufferHolder.buffer, 1, 
bufferHolder.totalSize())
--- End diff --

Why point to the same buffer after every iteration? We could do this during 
the construction of the iterator. ```BufferHolder``` might be overkill here, 
pointing to an array of 16 bytes should also do the trick.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12186] [WEB UI] Send the complete reque...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10180#issuecomment-165233362
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47839/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12376][TESTS] Spark Streaming Java8APIS...

[evanyc15]

GitHub user evanyc15 opened a pull request:

https://github.com/apache/spark/pull/10336

[SPARK-12376][TESTS] Spark Streaming Java8APISuite fails in 
assertOrderInvariantEquals method

org.apache.spark.streaming.Java8APISuite.java is failing due to trying to 
sort immutable list in assertOrderInvariantEquals method.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/evanyc15/spark 
SPARK-12376-StreamingJavaAPISuite

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10336.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10336


commit 28ff404c59aeff1b6604b70fb153d4d33e21635b
Author: Evan Chen 
Date:   2015-12-16T20:27:51Z

Changes to Spark Streaming Java8APISuite.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47829468
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -210,6 +210,37 @@ case class Sort(
   override def output: Seq[Attribute] = child.output
 }
 
+case class Range(
+start: Long,
+end: Long,
+step: Long,
+numSlices: Int) extends LeafNode {
+  require(step != 0, "step cannot be 0")
+  val numElements: BigInt = {
+val safeStart = BigInt(start)
+val safeEnd = BigInt(end)
+if ((safeEnd - safeStart) % step == 0 || safeEnd > safeStart ^ step > 
0) {
--- End diff --

I guess the reason is that `xor` is faster. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10335#issuecomment-165236262
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47852/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10335#issuecomment-165236260
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12376][TESTS] Spark Streaming Java8APIS...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10336#issuecomment-165236230
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10335#issuecomment-165236228
  
**[Test build #47852 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47852/consoleFull)**
 for PR 10335 at commit 
[`2aab4d6`](https://github.com/apache/spark/commit/2aab4d648fa634ec427e37ae82b0328fad159720).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Range(`\n  * `case class Range(`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/10335#issuecomment-165238312
  
The high level structure of this look pretty good to me.  Could you also 
post some numbers from a micro benchmark?  It would be good to make sure we're 
actually speeding things up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12364] [ML] [SparkR] Add ML example for...

[jkbradley]

Github user jkbradley commented on the pull request:

https://github.com/apache/spark/pull/10324#issuecomment-165242279
  
LGTM, and it worked when I tested it locally.
I'll go ahead and merge it with master and branch-1.6.
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11904] [PySpark] reduceByKeyAndWindow d...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9888#issuecomment-165222363
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12220][Core]Make Utils.fetchFile suppor...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/10208#issuecomment-165222160
  
Okey. I just removed my last 2 commits.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12220][Core]Make Utils.fetchFile suppor...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/10208#issuecomment-16557
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11904] [PySpark] reduceByKeyAndWindow d...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9888#issuecomment-165225457
  
**[Test build #47848 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47848/consoleFull)**
 for PR 9888 at commit 
[`6730f72`](https://github.com/apache/spark/commit/6730f72d2d9aa2c535abc9719e589369cc7b4cdb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11100][SQL] HiveThriftServer not regist...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/9113#issuecomment-165224194
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12220][Core]Make Utils.fetchFile suppor...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10208#issuecomment-165227079
  
**[Test build #47847 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47847/consoleFull)**
 for PR 10208 at commit 
[`2c31643`](https://github.com/apache/spark/commit/2c3164386040b5051e0332652cff9d2052b90cdb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10618] [Mesos] Refactoring scheduling c...

[tnachen]

Github user tnachen commented on the pull request:

https://github.com/apache/spark/pull/10326#issuecomment-165229364
  
Don't have anything else to add besides what @dragos said, but seems like 
it takes a while to get this updated. I vote for trying to merge this first as 
this adds more tests :+1: 
We can add finer grain logging later


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10325#issuecomment-165229352
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47831/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11749][Streaming] Duplicate creating th...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9765#issuecomment-165220327
  
**[Test build #47832 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47832/consoleFull)**
 for PR 9765 at commit 
[`625e20a`](https://github.com/apache/spark/commit/625e20aaccab8bb0102e5d465a326a289efef8bb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11749][Streaming] Duplicate creating th...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9765#issuecomment-165220591
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165224558
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47843/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10333#issuecomment-165224493
  
**[Test build #47843 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47843/consoleFull)**
 for PR 10333 at commit 
[`88b2107`](https://github.com/apache/spark/commit/88b21072b7e645075a31546a06edd8d5ea4d5176).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12220][Core]Make Utils.fetchFile suppor...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10208#issuecomment-165226020
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47846/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345] [CORE] Do not send SPARK_HOME th...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10329#issuecomment-165226015
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47845/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10618] [Mesos] Refactoring scheduling c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10326#issuecomment-165226384
  
**[Test build #47850 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47850/consoleFull)**
 for PR 10326 at commit 
[`f1bb5b1`](https://github.com/apache/spark/commit/f1bb5b1033216ac935d98a41e5528412dec13ec1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12345][MESOS] Filter SPARK_HOME when su...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10332#issuecomment-165227464
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47830/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/10335

[Spark-12374][SPARK-12150][SQL] Adding logical/physical operators for Range

Based on the suggestions from @marmbrus , added logical/physical operators 
for Range for improving the performance. 

Also added another API for resolving the JIRA Spark-12150. 

Could you take a look at my implementation, @marmbrus ? If not, I can 
rework it. : )

Thank you very much! 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark rangeOperators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10335.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10335


commit 2aab4d648fa634ec427e37ae82b0328fad159720
Author: gatorsmile 
Date:   2015-12-16T19:56:07Z

adding logical/physical operators for Range




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47830128
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -210,6 +210,37 @@ case class Sort(
   override def output: Seq[Attribute] = child.output
 }
 
+case class Range(
+start: Long,
+end: Long,
+step: Long,
+numSlices: Int) extends LeafNode {
+  require(step != 0, "step cannot be 0")
+  val numElements: BigInt = {
+val safeStart = BigInt(start)
+val safeEnd = BigInt(end)
+if ((safeEnd - safeStart) % step == 0 || safeEnd > safeStart ^ step > 
0) {
+  (safeEnd - safeStart) / step
+} else {
+  // the remainder has the same sign with range, could add 1 more
+  (safeEnd - safeStart) / step + 1
+}
+  }
+
+  val output: Seq[Attribute] =
+StructType(StructField("id", LongType, nullable = false) :: 
Nil).toAttributes
+  /**
+* Computes [[Statistics]] for this plan. The default implementation 
assumes the output
+* cardinality is the product of of all child plan's cardinality, i.e. 
applies in the case
+* of cartesian joins.
+*
+* [[LeafNode]]s must override this.
+*/
--- End diff --

You can omit scala doc that would be inherited.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-12374][SPARK-12150][SQL] Adding logical...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/10335#discussion_r47830108
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
 ---
@@ -210,6 +210,37 @@ case class Sort(
   override def output: Seq[Attribute] = child.output
 }
 
+case class Range(
+start: Long,
+end: Long,
+step: Long,
+numSlices: Int) extends LeafNode {
+  require(step != 0, "step cannot be 0")
+  val numElements: BigInt = {
+val safeStart = BigInt(start)
+val safeEnd = BigInt(end)
+if ((safeEnd - safeStart) % step == 0 || safeEnd > safeStart ^ step > 
0) {
+  (safeEnd - safeStart) / step
+} else {
+  // the remainder has the same sign with range, could add 1 more
+  (safeEnd - safeStart) / step + 1
+}
+  }
+
+  val output: Seq[Attribute] =
+StructType(StructField("id", LongType, nullable = false) :: 
Nil).toAttributes
+  /**
+* Computes [[Statistics]] for this plan. The default implementation 
assumes the output
+* cardinality is the product of of all child plan's cardinality, i.e. 
applies in the case
+* of cartesian joins.
+*
+* [[LeafNode]]s must override this.
+*/
+  val sizeInBytes = LongType.defaultSize * numElements
--- End diff --

`protected`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] Add missing interpolation in NettyRPCE...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10334#issuecomment-165247573
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47842/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] Add missing interpolation in NettyRPCE...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10334#issuecomment-165247566
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/9553#issuecomment-165249868
  
The use case here is that I want to build one common spark that is used 
across many different clusters.  Those clusters may not have Hive running yet, 
or perhaps I just don't want to have to specify hive-site.xml and include 
hcatalog, etc.  If I'm not using Hive then I don't need spark-shell to load it 
for me.  Perhaps I'm trying to use Spark and they take Hive down to do 
maintenance, now I can't run Spark or I get error messages out. 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12320][SQL] throw exception if the numb...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/10293#issuecomment-165249935
  
Okay, I'm going to merge this into master while removing the unnecessary 
deletion.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12153][MLlib]add support of arbitrary l...

[ygcao]

Github user ygcao commented on a diff in the pull request:

https://github.com/apache/spark/pull/10152#discussion_r47822243
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -281,17 +295,28 @@ class Word2Vec extends Serializable with Logging {
 val expTable = sc.broadcast(createExpTable())
 val bcVocab = sc.broadcast(vocab)
 val bcVocabHash = sc.broadcast(vocabHash)
-
-val sentences: RDD[Array[Int]] = words.mapPartitions { iter =>
+// each partition is a collection of sentences, will be translated 
into arrays of Index integer
+val sentences: RDD[Array[Int]] = dataset.mapPartitions { sentenceIter 
=>
   new Iterator[Array[Int]] {
-def hasNext: Boolean = iter.hasNext
+var wordIter: Iterator[String] = null
+
+def hasNext: Boolean = sentenceIter.hasNext || (wordIter != null 
&& wordIter.hasNext)
 
 def next(): Array[Int] = {
   val sentence = ArrayBuilder.make[Int]
   var sentenceLength = 0
-  while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
-val word = bcVocabHash.value.get(iter.next())
-word match {
+  // do translation of each word into its index in the vocabulary,
+  // do cutting only when the sentence is larger than 
maxSentenceLength
+  if ((wordIter == null || !wordIter.hasNext) && 
sentenceIter.hasNext) {
+do {
--- End diff --

This is just for the edge case when the input contains empty sentences, the 
do while loop will skip empty sentences instead of generate an empty array as a 
result for the empty sentence.
of course, if the sentence splitter will skip empty sentences, this is not 
necessary, but that's out of this class's control. So, I just recommend to have 
the loop, it won't introduce much latency since condition check is so cheap and 
the loop will only run once for 99.999% cases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org