[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-49388410 QA results for PR 1165:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds the following public classes (experimental):case class Sample(size: Lon

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-49383527 Oops, the tests will fail. We can't just set an initial memory threshold for some reason. Investigating. --- If your project is set up for it, you can reply to this e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-49383461 QA tests have started for PR 1165. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16797/consoleFull --- If

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15092309 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15092083 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15092024 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, max

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15091852 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -463,16 +463,16 @@ private[spark] class BlockManager( v

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15091634 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, max

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14846941 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14846875 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -67,10 +67,14 @@ class SparkEnv ( val metricsSystem: MetricsSystem, v

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14837879 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) ext

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14837855 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -67,10 +67,14 @@ class SparkEnv ( val metricsSystem: MetricsSystem, val

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14837775 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14837730 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14810095 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -67,10 +67,14 @@ class SparkEnv ( val metricsSystem: MetricsSystem, v

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14810039 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, max

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14810031 --- Diff: core/src/main/scala/org/apache/spark/util/collection/SizeTracker.scala --- @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809936 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809944 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, max

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809878 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809836 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -87,9 +97,32 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809809 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, max

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809771 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -463,16 +463,16 @@ private[spark] class BlockManager( v

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48701135 Did an initial pass with some feedback - not totally done yet but it should be enough to get some work done. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809244 --- Diff: project/MimaExcludes.scala --- @@ -61,16 +63,25 @@ object MimaExcludes { ProblemFilters.exclude[MissingMethodProblem](

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809203 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-11 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14809047 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808944 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808921 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808850 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808829 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808809 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -87,9 +97,32 @@ private class MemoryStore(blockManager: BlockManager, maxMemo

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808775 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -463,16 +463,16 @@ private[spark] class BlockManager( val

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808357 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -67,10 +67,14 @@ class SparkEnv ( val metricsSystem: MetricsSystem, val

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808344 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -67,10 +67,14 @@ class SparkEnv ( val metricsSystem: MetricsSystem, val

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808269 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) ext

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14808169 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) ext

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14802175 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -22,23 +22,37 @@ import java.util.LinkedHashMap import scala.coll

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14802119 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -22,23 +22,37 @@ import java.util.LinkedHashMap import scala.collec

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14801923 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14801587 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14801358 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -17,11 +17,13 @@ package org.apache.spark -import scala.collecti

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14801238 --- Diff: core/src/main/scala/org/apache/spark/util/collection/SizeTracker.scala --- @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14780352 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-10 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14780288 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, maxMe

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14739156 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14738375 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14738178 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14723528 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -38,7 +38,7 @@ import org.apache.spark.util._ private[spark] sealed t

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14723210 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14722303 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -87,9 +86,7 @@ private class MemoryStore(blockManager: BlockManager, maxMemory

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14722187 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14722107 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-09 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48502826 gmail was marking all mails from github as spam - apologies for the delay in getting to this ! Did not see the updates. --- If your project is set up for it, you can rep

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48424126 This is ready for further review. @pwendell @mateiz --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48421378 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48421379 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16437/ --- If your project

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48418619 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48418611 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48415357 I ran a few perf tests on 5 r3.8xlarge nodes, caching with both `MEMORY_ONLY` and `MEMORY_AND_DISK`, and observed no significant performance regression. In te

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48403666 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16425/ --- If your project

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48403664 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48398977 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-48398957 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14325606 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala --- @@ -50,6 +50,16 @@ class PrimitiveVector[@specialized(Long, Int, D

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14325602 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -87,9 +86,7 @@ private class MemoryStore(blockManager: BlockManager, maxMem

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14325588 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -55,8 +55,7 @@ private class MemoryStore(blockManager: BlockManager, maxMem

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14325571 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14325516 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-47432066 Looks like an interesting patch @andrewor14 ! I added a few comments based on a reasonably cursory glance. Will let someone else do a more detailed review. --- If

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14325021 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala --- @@ -50,6 +50,16 @@ class PrimitiveVector[@specialized(Long, Int, Doub

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14324995 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -87,9 +86,7 @@ private class MemoryStore(blockManager: BlockManager, maxMemory

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14324977 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -55,8 +55,7 @@ private class MemoryStore(blockManager: BlockManager, maxMemory

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14324964 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -61,9 +61,9 @@ private[spark] class BlockManager( // Actual stor

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14324910 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14324901 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14324897 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14324878 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) exte

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-28 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-47424883 Instead of trying to fit data into memory which cant be - isnt the current behavior of spark not better : simply fail if insufficient memory - user can always run with ME

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-47405235 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16221/ --- If your project

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-47405234 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-47401656 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-47401642 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46745371 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15981/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46745370 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46744819 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46744821 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46742967 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15979/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46742966 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46742154 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-46742149 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-06-20 Thread andrewor14
GitHub user andrewor14 opened a pull request: https://github.com/apache/spark/pull/1165 [SPARK-1777] Prevent OOMs from single partitions **Problem.** When caching, we currently unroll the entire RDD partition before making sure we have enough free memory. This is a common cause for

<    1   2