[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-07-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36018865 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -67,6 +68,8 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage

[GitHub] spark pull request: [Spark-6839] BlockManger.dataDeserialize must ...

2015-07-28 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/5463#issuecomment-125667585 sorry forgot to get back to this. Sure, I'll close. I think its not ideal that our leak-detector is just waiting for users to complain, but I also have no idea how

[GitHub] spark pull request: [Spark-6839] BlockManger.dataDeserialize must ...

2015-07-28 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/5463 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-8625] [Core] Propagate user exceptions ...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7014#discussion_r35658884 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -128,6 +149,25 @@ case class ExceptionFailure( } /** + * A class

[GitHub] spark pull request: [SPARK-8625] [Core] Propagate user exceptions ...

2015-07-28 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7014#issuecomment-125636498 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5259][CORE] don't submit stage until it...

2015-07-28 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7699#issuecomment-125554346 no problem @suyanNone . Sorry if I am being a little too eager on this one, but I just really want to get it into spark 1.5, and that deadline is coming up fast. You

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35692889 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -76,6 +76,37 @@ private[spark] abstract class Stage( */ private var

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35693474 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35693050 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -209,7 +209,7 @@ class DAGSchedulerSuite /** Send

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35692981 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -76,6 +76,37 @@ private[spark] abstract class Stage( */ private var

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35693396 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35693174 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35693973 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35694553 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35694635 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36297450 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -590,10 +590,21 @@ private[spark] class BlockManager( private def

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36297624 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -590,10 +590,21 @@ private[spark] class BlockManager( private def

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36297953 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -443,6 +448,34 @@ class BlockManagerSuite extends SparkFunSuite

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36298102 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -443,6 +448,34 @@ class BlockManagerSuite extends SparkFunSuite

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7927#issuecomment-127991678 thanks for updating @jeanlyn . Sorry that I didn't fully understand the issue earlier and for potentially changing the desired outcome on you. --- If your project

[GitHub] spark pull request: [SPARK-9384] [core] Easier setting of executor...

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7739#issuecomment-127991776 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9384] [core] Easier setting of executor...

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7739#issuecomment-127991794 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8366] When tasks failed and append new ...

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6817#issuecomment-127992407 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8366] When tasks failed and append new ...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6817#discussion_r36299283 --- Diff: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala --- @@ -787,6 +791,23 @@ class ExecutorAllocationManagerSuite

[GitHub] spark pull request: [SPARK-8366] When tasks failed and append new ...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6817#discussion_r36299328 --- Diff: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala --- @@ -787,6 +791,23 @@ class ExecutorAllocationManagerSuite

[GitHub] spark pull request: [SPARK-8366] When tasks failed and append new ...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6817#discussion_r36304518 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -628,6 +621,13 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-128014192 One other minor bit of housekeeping @ilganeli -- can you update the PR description, since that will go in the commit msg? I should probably also update the jira

[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service s...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7839#discussion_r36313470 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala --- @@ -0,0 +1,101 @@ +/* +* Licensed to the Apache

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7943#discussion_r36316371 --- Diff: network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java --- @@ -252,4 +269,30 @@ public String toString

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-128045237 Hi @tgravescs -- I know there is probably some cleanup to do here, but before getting into the weeds can I ask two high level questions: (1) Are you convinced

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7943#discussion_r36320303 --- Diff: network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java --- @@ -100,11 +127,34 @@ private boolean isAuthenticationEnabled

[GitHub] spark pull request: [SPARK-8029][core] shuffleoutput per attempt

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6648#issuecomment-128131451 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36260127 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,299 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36260666 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,299 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36261752 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,299 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36260830 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1190,10 +1193,17 @@ class DAGScheduler( if (errorMessage.isEmpty

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-127811976 thanks @ilganeli ! All of my comments are super minor. I think the cleanup on the tests makes things easier, we could probably even clean up some of the other tests

[GitHub] spark pull request: [SPARK-6880][CORE] Use properties from ActiveJ...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6291#discussion_r36262816 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -749,6 +752,43 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36260921 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -76,6 +77,35 @@ private[spark] abstract class Stage( */ private var

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127365342 Accumulators handle task failures, but have super confusing semantics around recomputation from shared lineage, speculative execution, and stage retries. (eg., I don't

[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service s...

2015-08-03 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7839#discussion_r36120783 --- Diff: network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java --- @@ -252,4 +264,20 @@ public String toString

[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service s...

2015-08-03 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7839#discussion_r36121140 --- Diff: network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java --- @@ -100,11 +119,33 @@ private boolean isAuthenticationEnabled

[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service s...

2015-08-03 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7839#discussion_r36120767 --- Diff: network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java --- @@ -252,4 +264,20 @@ public String toString

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127243422 I didn't realize the point of that other PR was so we could start switching all task metrics over ... accumulators have their own set of confusing semantics around

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36087907 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { Time that the executor spent

[GitHub] spark pull request: [SPARK-9437][core] avoid overflow in SizeEstim...

2015-07-30 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7750#issuecomment-126270650 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35694881 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35696484 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-07-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r35694369 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,319 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r35860163 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -85,6 +85,10 @@ private[spark] class Executor

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r35861096 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala --- @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r35861140 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala --- @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-126278222 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r35861272 --- Diff: core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala --- @@ -47,6 +53,27 @@ class NettyBlockTransferService(conf

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r35861374 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -175,20 +178,22 @@ private[spark] object SparkUI { listener

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r35861328 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -33,7 +33,8 @@ import org.apache.spark.TaskState.TaskState import

[GitHub] spark pull request: [SPARK-9104][CORE][WIP] expose Netty network l...

2015-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r35861382 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -175,20 +178,22 @@ private[spark] object SparkUI { listener

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-07 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36556494 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -590,10 +590,21 @@ private[spark] class BlockManager( private def

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-07 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36556522 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetchException.scala --- @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-8625] [Core] Propagate user exceptions ...

2015-08-07 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7014#issuecomment-128703873 @tomwhite still looks like a real compile error: ``` [error] /home/jenkins/workspace/NewSparkPullRequestBuilder/core/src/main/scala/org/apache/spark

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-07 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-128832922 Added the overall versioning serializing using json to avoid versioning problems from java object serialization (which also means I pull jackson into network/shuffle

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-07 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-128837771 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9809]Task crashes because the internal ...

2015-08-11 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8090#issuecomment-129862901 Thanks @carsonwang , I think I see -- it seems that the key part is that the stage is skipped (because the shuffle map output is computed by a different stage

[GitHub] spark pull request: [SPARK-9809]Task crashes because the internal ...

2015-08-11 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8090#issuecomment-129979596 so you don't think we should consider just completely skipping resetting the accumulators, and just always initializing the `internalAccumualtors` right away, since

[GitHub] spark pull request: [SPARK-9809]Task crashes because the internal ...

2015-08-11 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8090#issuecomment-130093251 Hi @andrewor14 , no that is not quite what I was saying. I was basically just repeating my comments from the earlier PR, and how I was confused on what we the goal

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-06 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-128523864 I've pushed an update to: * rely on yarn to call `stopApplication` after the NM comes back * write one executor per key in leveldb I still need to make

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36251038 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,299 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service s...

2015-08-04 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7839#issuecomment-127783165 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36250719 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,299 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36250802 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,299 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-6880][CORE] Use properties from ActiveJ...

2015-08-04 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6291#issuecomment-127779349 thanks mark, I should be able to look at this later tonight --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7943#discussion_r36481711 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala --- @@ -0,0 +1,193 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-05 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-128199952 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36405832 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,280 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-06 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/5636#issuecomment-128342749 lgtm! (last couple nits could even be fixed by the commiter) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36405882 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,280 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36405876 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,280 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36405856 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,280 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-9516][UI] Improvement of Thread Dump Pa...

2015-08-04 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7910#issuecomment-127686888 Hi @CodingCat just a quick question from looking at the screenshot -- can you still click on one thread to expand the stack trace for just that? nothing looks like

[GitHub] spark pull request: [SPARK-8366] When tasks failed and append new ...

2015-08-04 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6817#issuecomment-127684740 Hi @XuTingjun , I get multiple test failures in `ExecutorAllocationManagerSuite` when I try to run this locally, speficically listeners trigger add executors correctly

[GitHub] spark pull request: [SPARK-5945] Spark should not retry a stage in...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/5636#discussion_r36214635 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -473,6 +473,324 @@ class DAGSchedulerSuite

[GitHub] spark pull request: [SPARK-9516][UI] Improvement of Thread Dump Pa...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7910#discussion_r36438453 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala --- @@ -48,7 +48,17 @@ private[ui] class ExecutorThreadDumpPage(parent

[GitHub] spark pull request: [SPARK-8366] maxNumExecutorsNeeded should prop...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6817#discussion_r36439560 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -628,6 +621,13 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request: [SPARK-9092] Fixed incompatibility when both n...

2015-08-06 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7657#issuecomment-128459876 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9092] Fixed incompatibility when both n...

2015-08-06 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7657#issuecomment-128462332 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9092] Fixed incompatibility when both n...

2015-08-06 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7657#issuecomment-128462817 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7943#discussion_r36439043 --- Diff: network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java --- @@ -252,4 +294,18 @@ public String toString

[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6644#discussion_r36232759 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -57,6 +57,11 @@ class StorageStatus(val blockManagerId: BlockManagerId, val

[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...

2015-08-04 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-127734024 Hi @suyanNone any updates here? This is a really nice addition to help users understand when things go wrong, but needs a little followup. As Andrew requested, we'd

[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...

2015-08-04 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-127731343 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9384] [core] Easier setting of executor...

2015-08-07 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7739#issuecomment-128712008 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-07 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36557012 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -590,10 +590,24 @@ private[spark] class BlockManager( private def

[GitHub] spark pull request: [SPARK-9591][CORE]Job may fail for exception d...

2015-08-07 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7927#discussion_r36556866 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -592,8 +592,14 @@ private[spark] class BlockManager( val locations

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-13 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-130794558 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-13 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-130852333 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9924] [WEB UI] Don't schedule checkForL...

2015-08-13 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8153#issuecomment-130849109 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9809]Task crashes because the internal ...

2015-08-10 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/8090#discussion_r36708799 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -787,9 +787,10 @@ class DAGScheduler

[GitHub] spark pull request: [SPARK-9809]Task crashes because the internal ...

2015-08-10 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8090#issuecomment-129690335 Hi @carsonwang thanks for reporting this suggesting a fix. This is clearly really important to fix for 1.5, I'm glad you marked it as a blocker -- but I'd also like

[GitHub] spark pull request: [SPARK-9439] [wip] [yarn] External shuffle ser...

2015-08-14 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-131225100 hmm, last failure looks a bit fishy, but i've never observed that locally, and it passed on all the other tries on jenkins, so ... Jenkins, retest this please

[GitHub] spark pull request: [SPARK-9924] [WEB UI] Don't schedule checkForL...

2015-08-14 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8153#issuecomment-131197333 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

<    4   5   6   7   8   9   10   11   12   13   >