[GitHub] spark pull request: [SPARK-11661] [SQL] Still pushdown filters ret...

2015-11-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/9634#discussion_r44569414 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala --- @@ -236,8 +236,8 @@ abstract class BaseRelation { /** *

[GitHub] spark pull request: [SPARK-11267][WIP][Core]Fix the bug that Netty...

2015-11-11 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9237#issuecomment-155874897 I'm on Linux; that's really weird, because the Linux behavior is how I expect `SO_REUSEADDR` to work. From `man 7 socket`: When the listening socket is bound

[GitHub] spark pull request: [SPARK-11661] [SQL] Still pushdown filters ret...

2015-11-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/9634#discussion_r44569324 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -336,4 +336,29 @@ class ParquetFilterSuite

[GitHub] spark pull request: [SPARK-11661] [SQL] Still pushdown filters ret...

2015-11-11 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/9634 [SPARK-11661] [SQL] Still pushdown filters returned by unhandledFilters. https://issues.apache.org/jira/browse/SPARK-11661 You can merge this pull request into a Git repository by running: $ git

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9264#issuecomment-155874456 **[Test build #45660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45660/consoleFull)** for PR 9264 at commit [`0dd1128`](https://gith

[GitHub] spark pull request: [SPARK-9866][SQL] Speed up VersionsSuite by us...

2015-11-11 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9624#issuecomment-155874229 I don't really know what's the source of the problem. My best guess is some incompatibility between the version of ivy embedded in sbt and the version of ivy used by Spar

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread reggert
Github user reggert commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44568927 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -116,57 +119,26 @@ class SimpleFutureAction[T] private[spark](jobWaiter: JobWaiter[_]

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155873423 **[Test build #45661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45661/consoleFull)** for PR 9287 at commit [`40df523`](https://gith

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44568751 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -229,28 +183,15 @@ class ComplexFutureAction[T] extends FutureAction[T] {

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155872575 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread jacek-lewandowski
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155872516 jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155872618 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11262][ML] Unit test for gradient, loss...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9229#issuecomment-155872172 **[Test build #45658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45658/consoleFull)** for PR 9229 at commit [`083a9b3`](https://gith

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155871674 @yhuai RE your questions (3 was already addressed above): >1. Should we always ask users to provide pivot values? The argument for not requiring values I think is con

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread reggert
Github user reggert commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44568209 --- Diff: core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala --- @@ -66,14 +65,22 @@ class AsyncRDDActions[T: ClassTag](self: RDD[T]) extends Seri

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155870809 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155870813 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9264#issuecomment-155870734 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9264#issuecomment-155870704 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155870575 **[Test build #45659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45659/consoleFull)** for PR 7841 at commit [`12a8270`](https://gith

[GitHub] spark pull request: [SPARK-9866][SQL] Speed up VersionsSuite by us...

2015-11-11 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9624#issuecomment-155870366 Do you know what the origin of the problem is? Is it a concurrency or locking bug in Spark? There are many applications which use Ivy that don't run into these problem

[GitHub] spark pull request: [SPARK-11195][CORE] Use correct classloader fo...

2015-11-11 Thread brkyvz
Github user brkyvz commented on the pull request: https://github.com/apache/spark/pull/9367#issuecomment-155869908 My vote is with (3). I feel it requires the least amount of new code that you have to write and is more maintainable. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-11655] [core] Fix deadlock in handling ...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9633#issuecomment-155869442 **[Test build #45656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45656/consoleFull)** for PR 9633 at commit [`9fcd201`](https://gith

[GitHub] spark pull request: [SPARK-11644][SQL] Remove the option to turn o...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9618#issuecomment-155869238 **[Test build #45657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45657/consoleFull)** for PR 9618 at commit [`b296915`](https://gith

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44567088 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -116,57 +119,26 @@ class SimpleFutureAction[T] private[spark](jobWaiter: JobWaiter

[GitHub] spark pull request: [SPARK-11336] Add links to example codes

2015-11-11 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9320#issuecomment-155869334 @yinxusen Let's take a safe approach. Remove the link but leave the text: ~~~ Find full example code at examples/src/main/#{clean_markup} in the Spark repo.

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread aray
Github user aray commented on a diff in the pull request: https://github.com/apache/spark/pull/7841#discussion_r44566886 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/GroupedData.scala --- @@ -273,6 +280,60 @@ class GroupedData protected[sql]( def sum(colNames: Strin

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44567004 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -116,57 +119,26 @@ class SimpleFutureAction[T] private[spark](jobWaiter: JobWaiter

[GitHub] spark pull request: SPARK-2533 - Add locality levels on stage summ...

2015-11-11 Thread jbonofre
Github user jbonofre commented on the pull request: https://github.com/apache/spark/pull/9487#issuecomment-15586 Let me revert the unecessary commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project do

[GitHub] spark pull request: [SPARK-11267][WIP][Core]Fix the bug that Netty...

2015-11-11 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9237#issuecomment-155868674 > Is that true? This code throws an exception: What's your environment? I can run your codes without any exception. My OS is Mac OS X 10.10.5. --- If your proj

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9264#issuecomment-155868516 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155868392 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-11655] [core] Fix deadlock in handling ...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9633#issuecomment-155868363 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-11644][SQL] Remove the option to turn o...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9618#issuecomment-155868401 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9264#issuecomment-155868518 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [SPARK-11262][ML] Unit test for gradient, loss...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9229#issuecomment-155868424 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11262][ML] Unit test for gradient, loss...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9229#issuecomment-155868384 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-11655] [core] Fix deadlock in handling ...

2015-11-11 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9633#issuecomment-155868353 @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature ena

[GitHub] spark pull request: [SPARK-11644][SQL] Remove the option to turn o...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9618#issuecomment-155868367 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-11656][SQL] support typed aggregate in ...

2015-11-11 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/9630#discussion_r4450 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -381,6 +383,16 @@ class Dataset[T] private[sql]( new Dataset(sqlContext,

[GitHub] spark pull request: [SPARK-8992] [SQL] Add pivot to dataframe api

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7841#issuecomment-155868423 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11655] [core] Fix deadlock in handling ...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9633#issuecomment-155868405 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11655] [core] Fix deadlock in handling ...

2015-11-11 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/9633 [SPARK-11655] [core] Fix deadlock in handling of launcher stop(). The stop() callback was trying to close the launcher connection in the same thread that handles connection data, which ended up ca

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44566495 --- Diff: core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala --- @@ -95,19 +102,18 @@ class AsyncRDDActions[T: ClassTag](self: RDD[T]) extends

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44566403 --- Diff: core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala --- @@ -66,14 +65,22 @@ class AsyncRDDActions[T: ClassTag](self: RDD[T]) extends S

[GitHub] spark pull request: [SPARK-11419][STREAMING] Parallel recovery for...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9373#issuecomment-155867577 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [SPARK-11419][STREAMING] Parallel recovery for...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9373#issuecomment-155867574 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44566248 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -23,6 +23,7 @@ import java.util.concurrent.TimeUnit import java.uti

[GitHub] spark pull request: [SPARK-11419][STREAMING] Parallel recovery for...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9373#issuecomment-155867385 **[Test build #45648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45648/consoleFull)** for PR 9373 at commit [`1ba8340`](https://git

[GitHub] spark pull request: [SPARK-11656][SQL] support typed aggregate in ...

2015-11-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9630 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-11262][ML] Unit test for gradient, loss...

2015-11-11 Thread avulanov
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/9229#issuecomment-155867186 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155867075 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-11656][SQL] support typed aggregate in ...

2015-11-11 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/9630#issuecomment-155867131 Thanks, merging to master and 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155867110 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11262][ML] Unit test for gradient, loss...

2015-11-11 Thread avulanov
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/9229#issuecomment-155867077 @feynmanliang Thanks for the tip! My unit tests fail. It has a problem with the new RNG in Spark, so I updated the seeds and the asserts. However the fail seems to corr

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44565956 --- Diff: core/src/main/scala/org/apache/spark/scheduler/JobWaiter.scala --- @@ -49,29 +51,14 @@ private[spark] class JobWaiter[T]( dagScheduler.ca

[GitHub] spark pull request: [SQL][minor] rename present to finish in Aggre...

2015-11-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9617 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-10827] replace volatile with Atomic* in...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9611#issuecomment-155866709 **[Test build #2042 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2042/consoleFull)** for PR 9611 at commit [`b083309`](https://git

[GitHub] spark pull request: [SPARK-11646] WholeTextFileRDD should return T...

2015-11-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9622 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SQL][minor] rename present to finish in Aggre...

2015-11-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9617#issuecomment-155866425 I've merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-11646] WholeTextFileRDD should return T...

2015-11-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9622#issuecomment-155866549 (racing. I just merge this too) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-11646] WholeTextFileRDD should return T...

2015-11-11 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/9622#issuecomment-155866268 Merging this into master and 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10500][SPARKR] sparkr.zip cannot be cre...

2015-11-11 Thread brkyvz
Github user brkyvz commented on the pull request: https://github.com/apache/spark/pull/9390#issuecomment-155865783 @sun-rui With this patch, you should be able to enable this test: https://github.com/apache/spark/blob/27524a3a9ccee6fbe56149180ebfb3f74e0957e7/core/src/test/scala/org

[GitHub] spark pull request: [SPARK-11195][CORE] Use correct classloader fo...

2015-11-11 Thread choochootrain
Github user choochootrain commented on the pull request: https://github.com/apache/spark/pull/9367#issuecomment-155865241 @srowen @andrewor14 @yhuai it seems like the SPARK-8489 approach would be less invasive, but also less maintainable. any preferences? i'd like to stick wit

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9264#issuecomment-155864905 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9264#issuecomment-155864868 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread reggert
Github user reggert commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44564647 --- Diff: core/src/test/scala/org/apache/spark/rdd/AsyncRDDActionsSuite.scala --- @@ -197,4 +197,31 @@ class AsyncRDDActionsSuite extends SparkFunSuite with

[GitHub] spark pull request: [SPARK-11419][STREAMING] Parallel recovery for...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9373#issuecomment-155864120 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [SPARK-11419][STREAMING] Parallel recovery for...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9373#issuecomment-155864114 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-11419][STREAMING] Parallel recovery for...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9373#issuecomment-155864044 **[Test build #45650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45650/consoleFull)** for PR 9373 at commit [`1ba8340`](https://git

[GitHub] spark pull request: [SPARK-6328] [Python] Python API for Streaming...

2015-11-11 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9186#issuecomment-155864069 BTW, LGTM for the new Python APIs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-1267][PYSPARK] Adds pip installer for p...

2015-11-11 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/8318#discussion_r44564442 --- Diff: python/pyspark/__init__.py --- @@ -36,6 +36,53 @@ Finer-grained cache persistence levels. """ +import os +import re

[GitHub] spark pull request: SPARK-8398 hadoop input/output format advanced...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6848#issuecomment-155863800 **[Test build #45653 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45653/consoleFull)** for PR 6848 at commit [`208c019`](https://git

[GitHub] spark pull request: SPARK-8398 hadoop input/output format advanced...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6848#issuecomment-155863808 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: SPARK-8398 hadoop input/output format advanced...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6848#issuecomment-155863806 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-6328] [Python] Python API for Streaming...

2015-11-11 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9186#issuecomment-155863868 Could you add some codes in tests to demonstrate all fields in `JavaBatchInfo`, `JavaReceiverInfo` and `JavaOutputOperationInfo` can be accessed in Python? It's not nece

[GitHub] spark pull request: SPARK-8398 hadoop input/output format advanced...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6848#issuecomment-155863676 **[Test build #45653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45653/consoleFull)** for PR 6848 at commit [`208c019`](https://gith

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread reggert
Github user reggert commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44564235 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -20,13 +20,16 @@ package org.apache.spark import java.util.Collections impor

[GitHub] spark pull request: [SPARK-11373] [CORE] WiP Add metrics to the Hi...

2015-11-11 Thread CharlesYeh
Github user CharlesYeh commented on the pull request: https://github.com/apache/spark/pull/9571#issuecomment-155863566 I'm assuming you didn't make a HistorySource in HistoryServer (what MasterSource in Master does) because it requires the MetricsSystem. Where does MetricsSyst

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44564180 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -116,57 +120,26 @@ class SimpleFutureAction[T] private[spark](jobWaiter: JobWaiter

[GitHub] spark pull request: [SPARK-11644][SQL] Remove the option to turn o...

2015-11-11 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/9618#issuecomment-155863448 These two tests need fix: ``` org.apache.spark.sql.execution.metric.SQLMetricsSuite.Project metrics 42 ms 1 org.apache.spark.sql.hive.execution.HiveEx

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9632#issuecomment-155863384 **[Test build #45652 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45652/consoleFull)** for PR 9632 at commit [`8163966`](https://git

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9632#issuecomment-155863390 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9632#issuecomment-155863387 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9632#issuecomment-155862469 **[Test build #45652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45652/consoleFull)** for PR 9632 at commit [`8163966`](https://gith

[GitHub] spark pull request: [HOTFIX][SPARK-10192] Fix NPE in test that was...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9620#issuecomment-155862192 Oops, sorry. Thanks for fixing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44563390 --- Diff: core/src/test/scala/org/apache/spark/rdd/AsyncRDDActionsSuite.scala --- @@ -197,4 +197,31 @@ class AsyncRDDActionsSuite extends SparkFunSuite wit

[GitHub] spark pull request: [SPARK-6328] [Python] Python API for Streaming...

2015-11-11 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/9186#discussion_r44563303 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingListener.scala --- @@ -18,6 +18,82 @@ package org.apache.spark.strea

[GitHub] spark pull request: SPARK-8398 hadoop input/output format advanced...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6848#issuecomment-155861450 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-8398 hadoop input/output format advanced...

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6848#issuecomment-155861472 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11396] [SQL] add native implementation ...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9347#issuecomment-155861521 **[Test build #2041 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2041/consoleFull)** for PR 9347 at commit [`b2fca64`](https://git

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9632#issuecomment-155861473 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9632#issuecomment-155861439 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...

2015-11-11 Thread radekg
Github user radekg commented on a diff in the pull request: https://github.com/apache/spark/pull/9608#discussion_r44563045 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -810,7 +810,7 @@ private[spark] object Utils extends Logging { * Get the local ma

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9632#issuecomment-155860628 cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SQL][WIP] simplify encoder framework

2015-11-11 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/9632 [SQL][WIP] simplify encoder framework The idea is: keep the encoder unresolved until the last moment we need to use it for execution. Then we only need one method `bind` that need to be called

[GitHub] spark pull request: [SPARK-9026] Modifications to JobWaiter, Futur...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9264#discussion_r44562528 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -20,13 +20,16 @@ package org.apache.spark import java.util.Collections im

[GitHub] spark pull request: [SPARK-11638] [Mesos + Docker Bridge networkin...

2015-11-11 Thread radekg
Github user radekg commented on a diff in the pull request: https://github.com/apache/spark/pull/9608#discussion_r44562422 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -810,7 +810,7 @@ private[spark] object Utils extends Logging { * Get the local ma

[GitHub] spark pull request: [SPARK-11645][SQL] Remove OpenHashSet for the ...

2015-11-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9621#issuecomment-155859998 **[Test build #45651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45651/consoleFull)** for PR 9621 at commit [`bfdc937`](https://gith

[GitHub] spark pull request: SPARK-2533 - Add locality levels on stage summ...

2015-11-11 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9487#issuecomment-155859656 I agree, we should just put it under total time. The KV pairs at the top of the page are for summary info aggregated across tasks and that's exactly what this is. It

<    2   3   4   5   6   7   8   9   10   11   >