[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57762293 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57762288 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21242/consoleFull) for PR 2538 at commit [`e108ec1`](https://github.com/a

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57761606 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21241/consoleFull) for PR 2538 at commit [`d817de8`](https://github.com/a

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57761610 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3535][Mesos] Fix resource handling.

2014-10-02 Thread ash211
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/2401#issuecomment-57760828 This looks very reasonable. Counting the executor's bookkeeping core against the resources also seems much more correct than pretending it doesn't exist like before.

[GitHub] spark pull request: [SPARK-3535][Mesos] Fix resource handling.

2014-10-02 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/2401#discussion_r18382851 --- Diff: docs/configuration.md --- @@ -253,6 +253,17 @@ Apart from these, the following properties are also available, and may be useful spark.execut

[GitHub] spark pull request: [SPARK-3377] [SPARK-3610] Metrics can be accid...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2432#issuecomment-57760182 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3377] [SPARK-3610] Metrics can be accid...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2432#issuecomment-57760172 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21239/consoleFull) for PR 2432 at commit [`6570494`](https://github.com/a

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57758618 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57758615 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21237/consoleFull) for PR 2624 at commit [`ba77ca4`](https://github.com/a

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57758335 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21242/consoleFull) for PR 2538 at commit [`e108ec1`](https://github.com/ap

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57758118 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21241/consoleFull) for PR 2538 at commit [`d817de8`](https://github.com/ap

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57753844 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57753841 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21240/consoleFull) for PR 2633 at commit [`9dd7826`](https://github.com/a

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57753705 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21240/consoleFull) for PR 2633 at commit [`9dd7826`](https://github.com/ap

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread bijaybisht
Github user bijaybisht commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57753405 Fixed for Apache license headers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2778] [yarn] Add workaround for race in...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2605#issuecomment-57753034 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3377] [SPARK-3610] Metrics can be accid...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2432#issuecomment-57752832 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21239/consoleFull) for PR 2432 at commit [`6570494`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3377] [SPARK-3610] Metrics can be accid...

2014-10-02 Thread sarutak
Github user sarutak commented on a diff in the pull request: https://github.com/apache/spark/pull/2432#discussion_r18381560 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala --- @@ -98,10 +96,45 @@ private[spark] class MetricsSystem private ( sinks

[GitHub] spark pull request: [SPARK-2778] [yarn] Add workaround for race in...

2014-10-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2605#issuecomment-57751730 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57751658 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21237/consoleFull) for PR 2624 at commit [`ba77ca4`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3377] [SPARK-3610] Metrics can be accid...

2014-10-02 Thread sarutak
Github user sarutak commented on a diff in the pull request: https://github.com/apache/spark/pull/2432#discussion_r18381532 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -187,6 +187,15 @@ class SparkContext(config: SparkConf) extends Logging { val m

[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...

2014-10-02 Thread manishamde
Github user manishamde commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-57751031 @chouqin Yes, separate PR sounds good. Let's remember to create a JIRA. We can also try to estimate it based upon the Spark conf and dataset size. --- If your projec

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2501#issuecomment-57748176 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2501#issuecomment-57748170 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21236/consoleFull) for PR 2501 at commit [`0ea889e`](https://github.com/a

[GitHub] spark pull request: [SPARK-3771][SQL] AppendingParquetOutputFormat...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2638#issuecomment-57746994 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/259/consoleFull) for PR 2638 at commit [`ec213c1`](https://github.com/

[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...

2014-10-02 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/2616#issuecomment-57746732 Comment out the print statement in merge_pr that causes the failure. On Thu, Oct 2, 2014 at 7:58 PM, Michael Armbrust wrote: > Tried merging bu

[GitHub] spark pull request: [SPARK-3654][SQL] Implement all extended HiveQ...

2014-10-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2590 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3654][SQL] Implement all extended HiveQ...

2014-10-02 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/2590#issuecomment-57746419 Thanks! I've merged this to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...

2014-10-02 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/2616#issuecomment-57745439 Tried merging but it failed :( @kayousterhout what did you end up doing to merge this the first time? --- If your project is set up for it, you can reply to thi

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread giwa
Github user giwa commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18379709 --- Diff: python/pyspark/streaming/dstream.py --- @@ -0,0 +1,624 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contri

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2501#issuecomment-57744198 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21236/consoleFull) for PR 2501 at commit [`0ea889e`](https://github.com/ap

[GitHub] spark pull request: [SPARK-2778] [yarn] Add workaround for race in...

2014-10-02 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/2605#issuecomment-57742306 I took a look and found where the behavior that Marcelo is observing is occurring in YARN. (For future reference), when the ResourceManager's ClientRMService starts up, it

[GitHub] spark pull request: [SQL] Initilize session state before creating ...

2014-10-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2635 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3771][SQL] AppendingParquetOutputFormat...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2638#issuecomment-57741192 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/259/consoleFull) for PR 2638 at commit [`ec213c1`](https://github.com/a

[GitHub] spark pull request: [SPARK-3774] typo comment in bin/utils.sh

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2639#issuecomment-57740737 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-3774] typo comment in bin/utils.sh

2014-10-02 Thread tsudukim
GitHub user tsudukim opened a pull request: https://github.com/apache/spark/pull/2639 [SPARK-3774] typo comment in bin/utils.sh Modified the comment of bin/utils.sh. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tsudukim/spark f

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-57740477 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-57740476 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21235/consoleFull) for PR 2497 at commit [`61162a6`](https://github.com/a

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2501#issuecomment-57740051 LGTM except for some minor issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18378532 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/api/python/PythonDStream.scala --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2501#discussion_r18378504 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala --- @@ -17,22 +17,61 @@ package org.apache.spark.sql.hive

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2501#discussion_r18378483 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -305,7 +300,7 @@ private[hive] case class MetastoreRelation

[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-57739363 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-57739361 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21234/consoleFull) for PR 2595 at commit [`db0d24a`](https://github.com/a

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18378352 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala --- @@ -52,12 +52,18 @@ private[spark] class PythonRDD( accumulator: Accu

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2501#discussion_r18378336 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala --- @@ -428,7 +429,7 @@ class TestHiveContext(sc: SparkContext) extends HiveCont

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2501#discussion_r18378326 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/CacheManager.scala --- @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57739142 Hmm, strange test failure: ``` [info] - block generator throttling *** FAILED *** [info] org.scalatest.exceptions.TestFailedException was thrown. (Net

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57738844 Seems like the unit test are failing with GC-overhead-too-high errors. @davies Please take a look. I can help out if you want. --- If your project is set up for it, you ca

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2469#issuecomment-57738768 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21233/consoleFull) for PR 2469 at commit [`d121883`](https://github.com/a

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2469#issuecomment-57738773 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18378148 --- Diff: python/pyspark/streaming/dstream.py --- @@ -0,0 +1,624 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contri

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18378110 --- Diff: python/pyspark/streaming/context.py --- @@ -0,0 +1,305 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contri

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18378095 --- Diff: python/pyspark/streaming/context.py --- @@ -0,0 +1,305 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contri

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2501#discussion_r18377923 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/SameResultSuite.scala --- @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-3138][SQL] sqlContext.parquetFile shoul...

2014-10-02 Thread mohnishkodnani
Github user mohnishkodnani commented on the pull request: https://github.com/apache/spark/pull/2044#issuecomment-57737832 Can we have this take a parent directory that has other subdirectories with parquet files, instead of the parquet files being the immediate children of the directo

[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...

2014-10-02 Thread chouqin
Github user chouqin commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-57737757 @manishamde I think `maxMemoryInMB` is now setting for executors. Executor's memory is by default bigger than driver's memory(1G vs 512M), but it also needs to store a pa

[GitHub] spark pull request: [DEPLOY] SPARK-3759: Return the exit code of t...

2014-10-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2628 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [DEPLOY] SPARK-3759: Return the exit code of t...

2014-10-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2628#issuecomment-57736057 Thanks, merging this into master and 1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proje

[GitHub] spark pull request: [SPARK-3755][Core] avoid trying privileged por...

2014-10-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2623#issuecomment-57734982 Hey @scwf can you update the title and the description in the JIRA? It's now inconsistent with what this PR actually does. --- If your project is set up for it, you c

[GitHub] spark pull request: [SPARK-3713][SQL] Uses JSON to serialize DataT...

2014-10-02 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2563#discussion_r18377127 --- Diff: python/pyspark/sql.py --- @@ -62,6 +63,12 @@ def __eq__(self, other): def __ne__(self, other): return not self.__eq__(other)

[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...

2014-10-02 Thread manishamde
Github user manishamde commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-57734254 @chouqin I just took a look at the PR. This is excellent. I don't have any more comments to add apart from what has been discussed already. Very interested to

[GitHub] spark pull request: [SPARK-2750] support https in spark web ui

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57734206 I followed the instructions at http://stackoverflow.com/a/14408381/590203 to generate a self-signed certificate, but unfortunately I see errors when I try to browse to

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-57733873 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21235/consoleFull) for PR 2497 at commit [`61162a6`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3755][Core] avoid trying privileged por...

2014-10-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2623 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3755][Core] avoid trying privileged por...

2014-10-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2623#issuecomment-57733466 Ok, merging this again into master and 1.1. Thanks @scwf. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57733269 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21232/consoleFull) for PR 2624 at commit [`ee62bb7`](https://github.com/a

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57733279 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-57733203 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21234/consoleFull) for PR 2595 at commit [`db0d24a`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...

2014-10-02 Thread chouqin
Github user chouqin commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-57733146 @jkbradley thanks for your comments, I have changed my code, could you please have a look? --- If your project is set up for it, you can reply to this email and have you

[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...

2014-10-02 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2616#issuecomment-57733000 @marmbrus Let's try to merge this one to master and see whether Jenkins accepts it. --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2469#issuecomment-57732513 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21233/consoleFull) for PR 2469 at commit [`d121883`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3713][SQL] Uses JSON to serialize DataT...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2563#issuecomment-57732079 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21227/consoleFull) for PR 2563 at commit [`81e28fb`](https://github.com/a

[GitHub] spark pull request: [SPARK-3713][SQL] Uses JSON to serialize DataT...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2563#issuecomment-57732086 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3755][Core] avoid trying privileged por...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2623#issuecomment-57731789 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3755][Core] avoid trying privileged por...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2623#issuecomment-57731785 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21228/consoleFull) for PR 2623 at commit [`10a4437`](https://github.com/a

[GitHub] spark pull request: [SPARK-2750] support https in spark web ui

2014-10-02 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57731222 Ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wis

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2501#issuecomment-57731180 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2501#issuecomment-57731178 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21229/consoleFull) for PR 2501 at commit [`1e23287`](https://github.com/a

[GitHub] spark pull request: [SPARK-2750] support https in spark web ui

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57731094 The table in the documentation should include `spark.ui.https.enabled`, too; looks like it's missing now, which was confusing. --- If your project is set up for it, yo

[GitHub] spark pull request: [SPARK-2750] support https in spark web ui

2014-10-02 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57730857 Yes, here i did not change it, because "http://joshs-mbp:4041"; will redirect to https url --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: [SPARK-2750] support https in spark web ui

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57730551 I'm testing this out locally with my own keystore and truststore. Still debugging some issues (maybe I've misconfigured my keys), but I noticed that even with HTTPS en

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57730092 We should probably update the docstrings to remove all references to ThreadLocal, too. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-3771][SQL] AppendingParquetOutputFormat...

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2638#issuecomment-57729963 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-10-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1222#issuecomment-57729739 Yes - I mention the need for a read API for this in my comment above, if we want this to be public. The current directory name is used as the final log name, but w

[GitHub] spark pull request: [SPARK-3762] clear reference of SparkEnv after...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2624#issuecomment-57729754 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21232/consoleFull) for PR 2624 at commit [`ee62bb7`](https://github.com/ap

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57729657 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-57729651 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21226/consoleFull) for PR 2538 at commit [`37fe06f`](https://github.com/a

[GitHub] spark pull request: [SPARK-3771][SQL] AppendingParquetOutputFormat...

2014-10-02 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/2638 [SPARK-3771][SQL] AppendingParquetOutputFormat should use reflection to prevent breaking binary-compatibility. Original problem is [SPARK-3764](https://issues.apache.org/jira/browse/SPARK-3764).

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57727917 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57727911 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21230/consoleFull) for PR 2633 at commit [`2002151`](https://github.com/a

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57727877 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21230/consoleFull) for PR 2633 at commit [`2002151`](https://github.com/ap

[GitHub] spark pull request: Event proration based on event timestamps.

2014-10-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2633#issuecomment-57727001 ok to test. Hey @bijaybisht can you open an associated JIRA for this PR? See how other PRs are opened. --- If your project is set up for it, you can reply to this ema

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-10-02 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1222#issuecomment-57726091 Hey @vanzin I will take a more detailed look shortly. From a quick glance it seems that now we encode the metadata in the first part of the log file and use a header m

[GitHub] spark pull request: [SPARK-3212][SQL] Use logical plan matching in...

2014-10-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2501#issuecomment-57726136 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21229/consoleFull) for PR 2501 at commit [`1e23287`](https://github.com/ap

[GitHub] spark pull request: [SPARK-2750] support https in spark web ui

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57725962 Hey, sorry for the delay; I've been swamped with other work. I'm going to take a look right now... --- If your project is set up for it, you can reply to this email a

[GitHub] spark pull request: [SPARK-2750] support https in spark web ui

2014-10-02 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1980#issuecomment-57725678 Have added redirect from http to https and updated docs. Some directions as follows 1 There is no changes for web ui when not set ```spark.ui.https.enabled``` 2 When

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18374241 --- Diff: python/pyspark/streaming/dstream.py --- @@ -0,0 +1,624 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# c

[GitHub] spark pull request: [SPARK-2377] Python API for Streaming

2014-10-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18374226 --- Diff: python/pyspark/streaming/dstream.py --- @@ -0,0 +1,624 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# c

  1   2   3   4   5   >