[GitHub] spark pull request #19287: [SPARK-22074][Core] Task killed by other attempt ...

2017-09-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19287#discussion_r141643992 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -724,6 +724,7 @@ private[spark] class TaskSetManager

[GitHub] spark pull request #19287: [SPARK-22074][Core] Task killed by other attempt ...

2017-09-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19287#discussion_r141668660 --- Diff: core/src/test/scala/org/apache/spark/scheduler/FakeTask.scala --- @@ -58,4 +57,18 @@ object FakeTask { } new TaskSet(tasks

[GitHub] spark pull request #19287: [SPARK-22074][Core] Task killed by other attempt ...

2017-09-28 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19287#discussion_r141674264 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -744,6 +744,100 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 Sorry if I was unclear earlier on the issue w/ the active Job ID. So, I agree, that if a user actually gets into this situation, where they've got two different jobs for the same stage

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r141475205 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -512,6 +535,9 @@ private[spark] class TaskSetManager

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 btw I'm not opposed to this feature, just want to make sure that given (a) complexity and (b) the fact that it doesn't quite meet the original ask anyway (configuring at the stage & no need to b

[GitHub] spark issue #19343: [SPARK-22121][SQL] Correct database location for namenod...

2017-09-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19343 OK, closing this and the jira --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #19343: [SPARK-22121][SQL] Correct database location for ...

2017-09-26 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/19343 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps

2017-09-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19250 @HyukjinKwon you might be interested in this one also --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19343: [SPARK-22121][SQL] Correct database location for namenod...

2017-09-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19343 I don't see much point in putting this in the docs ... it seems too fine-grained a detail to be useful there. I just don't see the users who encounter this exception from going to look at the spot

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 one more thought -- have you considered adding stage properties? I know that is more involved since its an API change, but I dunno how many times I've wanted something like that, eg. for labelling

[GitHub] spark pull request #19338: [SPARK-22123][CORE] Add latest failure reason for...

2017-09-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19338#discussion_r141098993 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala --- @@ -61,6 +61,16 @@ private[scheduler] class TaskSetBlacklist(val conf

[GitHub] spark issue #19343: [SPARK-22121][SQL] Correct database location for namenod...

2017-09-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19343 whoops, sorry I wrote [CORE] out of habit! > Spark SQL might not be deployed in the HDFS system. Conceptually, this HDFS-specific codes should not be part of our HiveExternalCata

[GitHub] spark pull request #19343: [SPARK-22121][CORE] Correct database location for...

2017-09-25 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19343 [SPARK-22121][CORE] Correct database location for namenode HA. ## What changes were proposed in this pull request? If hdfs HA is turned on after a hive database is already created, the db

[GitHub] spark pull request #19331: [SPARK-22109][SQL] Resolves type conflicts betwee...

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19331#discussion_r140890429 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala --- @@ -470,15 +471,15 @@ object PartitioningUtils

[GitHub] spark pull request #19338: [SPARK-21539][CORE] Add latest failure reason for...

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19338#discussion_r140823560 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala --- @@ -94,7 +96,9 @@ private[scheduler] class TaskSetBlacklist(val conf

[GitHub] spark pull request #19338: [SPARK-21539][CORE] Add latest failure reason for...

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19338#discussion_r140823764 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -838,7 +840,7 @@ private[spark] class TaskSetManager

[GitHub] spark issue #19338: [SPARK-21539][CORE] Add latest failure reason for task s...

2017-09-25 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19338 @caneGuy thanks for working on this, looks very reasonable to me, I am going to take a closer look at a couple of details. But can you make a couple of updates in the meantime: 1) Can you

[GitHub] spark issue #19338: [SPARK-21539][CORE] Add latest failure reason for task s...

2017-09-25 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19338 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19311#discussion_r140815961 --- Diff: core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala --- @@ -407,4 +407,119 @@ class MemoryStoreSuite }) assert

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19311#discussion_r140813788 --- Diff: core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala --- @@ -407,4 +407,119 @@ class MemoryStoreSuite }) assert

[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18853#discussion_r140802924 --- Diff: docs/sql-programming-guide.md --- @@ -1460,6 +1460,13 @@ that these options will be deprecated in future release as more optimizations ar

[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18853#discussion_r140799432 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -115,21 +115,46 @@ object TypeCoercion

[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18853#discussion_r140801036 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2677,4 +2677,142 @@ class SQLQuerySuite extends QueryTest

[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...

2017-09-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18853#discussion_r140797299 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -2677,4 +2677,142 @@ class SQLQuerySuite extends QueryTest

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19311#discussion_r140559549 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -544,20 +544,39 @@ private[spark] class MemoryStore

[GitHub] spark pull request #19313: [SPARK-21928][CORE] Set classloader on Serializer...

2017-09-21 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/19313 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19313: [SPARK-21928][CORE] Set classloader on SerializerManager...

2017-09-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19313 thanks @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19313: [SPARK-21928][CORE] Set classloader on Serializer...

2017-09-21 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19313 [SPARK-21928][CORE] Set classloader on SerializerManager's private kryo ## What changes were proposed in this pull request? We have to make sure that SerializerManager's private instance

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r140337893 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-21 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19311 [SPARK-22083][CORE] Release locks in MemoryStore.evictBlocksToFreeSpace ## What changes were proposed in this pull request? MemoryStore.evictBlocksToFreeSpace acquires write locks for all

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r140296863 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark issue #19280: [SPARK-21928][CORE] Set classloader on SerializerManager...

2017-09-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19280 > Looks ok to me, assuming the "default serializer" in SerializerManager is configured correctly through other means. I think that part is fine. The serializer is created her

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r140136101 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark issue #19280: [SPARK-21928][CORE] Set classloader on SerializerManager...

2017-09-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19280 reaching out to some potential reviewers: @vanzin @srowen @JoshRosen @mridulm @tgravescs --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps

2017-09-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19250 also cc @yhuai @liancheng would appreciate a review since you've looked at sql & hive compatibility in the

[GitHub] spark issue #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory usage to...

2017-09-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19160 lgtm @jerryshao I didn't merge yet in case you want to keep discussing the naming, but I'm fine with this as is, so feel free to merge

[GitHub] spark pull request #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory u...

2017-09-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19160#discussion_r140051678 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -248,6 +251,16 @@ private[spark] class BlockManager( logInfo(s

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r139819360 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -1255,6 +1255,97 @@ class TaskSetManagerSuite extends

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r139819793 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -1255,6 +1255,97 @@ class TaskSetManagerSuite extends

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r139821899 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r139818031 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -758,11 +825,52 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark issue #19145: [spark-21933][yarn] Spark Streaming request more executo...

2017-09-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19145 I'm not sure I totally follow the sequence of events, but I get the feeling this should be handled in yarn, not spark. Also, I agree with Jerry, it seems like your `completedContainerIdSet

[GitHub] spark pull request #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory u...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19160#discussion_r139800662 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -115,6 +115,7 @@ private[spark] class Executor( if (!isLocal

[GitHub] spark pull request #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory u...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19160#discussion_r139796177 --- Diff: core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala --- @@ -18,11 +18,14 @@ package

[GitHub] spark pull request #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory u...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19160#discussion_r139795417 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -248,6 +251,16 @@ private[spark] class BlockManager( logInfo(s

[GitHub] spark pull request #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory u...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19160#discussion_r139799940 --- Diff: core/src/main/scala/org/apache/spark/deploy/ExternalShuffleServiceSource.scala --- @@ -19,19 +19,19 @@ package org.apache.spark.deploy

[GitHub] spark issue #19211: [SPARK-18838][core] Add separate listener queues to Live...

2017-09-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19211 lgtm do you know how much better this makes it? Eg., if we had an existing case where things go haywire in dynamic allocation because of this -- we could see if after this change, the only

[GitHub] spark pull request #19211: [SPARK-18838][core] Add separate listener queues ...

2017-09-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19211#discussion_r139787945 --- Diff: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala --- @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19280: [SPARK-21928][CORE] Set classloader on Serializer...

2017-09-19 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19280 [SPARK-21928][CORE] Set classloader on SerializerManager private kryo ## What changes were proposed in this pull request? We have to make sure thatthat SerializerManager's private instance

[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps

2017-09-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19250 Hi @ueshin @rxin, could you please review? thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-09-15 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19250#discussion_r139234142 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala --- @@ -92,7 +92,7 @@ case class

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-09-15 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19250 [SPARK-12297] Table timezone correction for Timestamps ## What changes were proposed in this pull request? When reading and writing data, spark will adjust timestamp data based

[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark

2017-09-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18935 lgtm any more thoughts @jiangxb1987 @zsxwing ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #16992: [SPARK-19662][SCHEDULER][TEST] Add Fair Scheduler Unit T...

2017-08-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16992 merged to master thanks @erenavsarogullari , sorry again for the delays --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r134501736 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -724,6 +777,62 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r134498890 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -724,6 +777,62 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r134501323 --- Diff: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala --- @@ -188,10 +192,195 @@ class ExecutorAllocationManagerSuite

[GitHub] spark issue #18950: [SPARK-20589][Core][Scheduler] Allow limiting task concu...

2017-08-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18950 @dhruve looks like a real test failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark

2017-08-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18935 thanks for the added info @jerryshao. (a) ok makes sense now about the use of this -- it is not exposed now, you plan to expose it in future changes. That is fine, however I would like

[GitHub] spark issue #18887: [SPARK-20642][core] Store FsHistoryProvider listing data...

2017-08-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18887 I futzed around for a while with trying to keep the old stuff around, and I realized it really would be quite a mess. The biggest problem is that the old rest api is just waay too tied into the UI

[GitHub] spark issue #16992: [SPARK-19662][SCHEDULER][TEST] Add Fair Scheduler Unit T...

2017-08-18 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16992 been a while so lets run tests again just to check: Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #16992: [SPARK-19662][SCHEDULER][TEST] Add Fair Scheduler...

2017-08-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16992#discussion_r134056863 --- Diff: docs/job-scheduling.md --- @@ -235,7 +235,7 @@ properties: of the cluster. By default, each pool's `minShare` is 0. The pool

[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...

2017-08-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18935#discussion_r134052216 --- Diff: common/network-common/src/main/java/org/apache/spark/network/util/NettyMemoryMetrics.java --- @@ -0,0 +1,145 @@ +/* + * Licensed

[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...

2017-08-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18935#discussion_r134051537 --- Diff: common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java --- @@ -106,6 +105,12 @@ public

[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...

2017-08-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18935#discussion_r134050644 --- Diff: common/network-common/src/main/java/org/apache/spark/network/util/NettyMemoryMetrics.java --- @@ -0,0 +1,145 @@ +/* + * Licensed

[GitHub] spark issue #18887: [SPARK-20642][core] Store FsHistoryProvider listing data...

2017-08-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18887 Just to keep others in the loop, Marcelo and I talked about this some offline. I think this PR itself is fine, but to me this is an important point in the larger history server project he's doing

[GitHub] spark pull request #18887: [SPARK-20642][core] Store FsHistoryProvider listi...

2017-08-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18887#discussion_r133725104 --- Diff: scalastyle-config.xml --- @@ -86,7 +86,7 @@ This file is divided into 3 sections: - + --- End diff

[GitHub] spark issue #18887: [SPARK-20642][core] Store FsHistoryProvider listing data...

2017-08-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18887 >> I feel like we should have one version where the old code is still available, controlled by a feature flag. >I'm not sure exactly what you're suggesting. The default behavior

[GitHub] spark pull request #18887: [SPARK-20642][core] Store FsHistoryProvider listi...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18887#discussion_r133512286 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -316,25 +350,22 @@ private[history] class FsHistoryProvider

[GitHub] spark pull request #18887: [SPARK-20642][core] Store FsHistoryProvider listi...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18887#discussion_r133517697 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -742,53 +696,146 @@ private[history] object FsHistoryProvider

[GitHub] spark pull request #18887: [SPARK-20642][core] Store FsHistoryProvider listi...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18887#discussion_r133510977 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -301,6 +334,7 @@ private[history] class FsHistoryProvider(conf

[GitHub] spark pull request #18887: [SPARK-20642][core] Store FsHistoryProvider listi...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18887#discussion_r133511528 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -117,17 +122,37 @@ private[history] class FsHistoryProvider

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r133505348 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -1214,6 +1214,101 @@ class TaskSetManagerSuite extends

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r133504041 --- Diff: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala --- @@ -188,6 +188,125 @@ class ExecutorAllocationManagerSuite

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r133504438 --- Diff: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala --- @@ -188,6 +188,125 @@ class ExecutorAllocationManagerSuite

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r133502797 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -314,7 +316,7 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r133499828 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -602,6 +604,21 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #18887: [SPARK-20642][core] Store FsHistoryProvider listi...

2017-08-15 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18887#discussion_r133316362 --- Diff: scalastyle-config.xml --- @@ -86,7 +86,7 @@ This file is divided into 3 sections: - + --- End diff

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-15 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r133311545 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -602,6 +604,21 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-15 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18950#discussion_r133311882 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -454,64 +477,68 @@ private[spark] class TaskSetManager

[GitHub] spark issue #18874: [SPARK-21656][CORE] spark dynamic allocation should not ...

2017-08-15 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18874 @srowen you have a good point about a case that becomes worse after this change. Still I think this change is better on balance. btw, there are more even more odd cases with dynamic

[GitHub] spark issue #18874: [SPARK-21656][CORE] spark dynamic allocation should not ...

2017-08-10 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18874 This change makes sense to me. Tom's last comment about resetting that timeout every time one task is scheduled I think explains how you get in this situation and why you don't actually

[GitHub] spark pull request #18739: [WIP][SPARK-21539][CORE] Job should not be aborte...

2017-07-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18739#discussion_r129849052 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -665,10 +667,15 @@ private[spark] class TaskSetManager

[GitHub] spark pull request #18739: [WIP][SPARK-21539][CORE] Job should not be aborte...

2017-07-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18739#discussion_r129749281 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -644,10 +644,12 @@ private[spark] class TaskSetManager

[GitHub] spark pull request #18739: [WIP][SPARK-21539][CORE] Job should not be aborte...

2017-07-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18739#discussion_r129749722 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -665,10 +667,15 @@ private[spark] class TaskSetManager

[GitHub] spark issue #17902: [SPARK-20641][core] Add key-value store abstraction and ...

2017-06-06 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17902 merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17902: [SPARK-20641][core] Add key-value store abstraction and ...

2017-06-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17902 fair enough on jackson. add the log for the seed please, other than that, lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #17113: [SPARK-13669][SPARK-20898][Core] Improve the blacklist m...

2017-06-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17113 I'm fine with this, just a couple asks: 1) is there a test for SPARK-20898? I know at some point in the development of killing blacklisted executors we tested on yarn, I'm really

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-06-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119904509 --- Diff: common/kvstore/src/test/java/org/apache/spark/kvstore/DBIteratorSuite.java --- @@ -0,0 +1,494 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-06-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119652958 --- Diff: common/kvstore/src/test/java/org/apache/spark/kvstore/DBIteratorSuite.java --- @@ -0,0 +1,500 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-06-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119657514 --- Diff: common/kvstore/src/test/java/org/apache/spark/kvstore/LevelDBTypeInfoSuite.java --- @@ -0,0 +1,192 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-06-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119651685 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDBTypeInfo.java --- @@ -0,0 +1,502 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-06-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119649224 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDBTypeInfo.java --- @@ -0,0 +1,502 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-05-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119408662 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/KVStoreView.java --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-05-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119409655 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/KVTypeInfo.java --- @@ -0,0 +1,154 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-05-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119411351 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/KVStore.java --- @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-05-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119420304 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDB.java --- @@ -0,0 +1,303 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-05-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119407982 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/KVStoreView.java --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-05-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r119412048 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDB.java --- @@ -0,0 +1,303 @@ +/* + * Licensed to the Apache Software

<    7   8   9   10   11   12   13   14   15   16   >