Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/147#discussion_r10635968
--- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
---
@@ -136,4 +142,30 @@ class MapOutputTrackerSuite extends FunSuite
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/147#issuecomment-37743968
This should be ready to merge unless other people have more to add.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user andrewor14 closed the pull request at:
https://github.com/apache/spark/pull/147
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/147#issuecomment-37744167
Continued at #152. Closing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616395
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -22,24 +22,25 @@ import java.text.SimpleDateFormat
import java.util
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616539
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerStatusListener.scala
---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616549
--- Diff: core/src/main/scala/org/apache/spark/storage/PutResult.scala ---
@@ -20,7 +20,13 @@ package org.apache.spark.storage
import
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616570
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -196,13 +152,46 @@ class DAGScheduler
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616671
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +105,53 @@ private[spark] class SparkUI(sc: SparkContext) extends
Logging
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/147
[SPARK-1244] Throw exception if map output status exceeds frame size
In the existing code, this fails silently...
You can merge this pull request into a Git repository by running:
$ git pull
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10633389
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -80,187 +81,78 @@ class JobLogger(val user: String, val logDirName:
String
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10633506
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/89#issuecomment-37554932
@mridulm and @puravaggarwal Thanks for doing this. As a general direction,
however, we are actually moving away from MetadataCleaner in the long run -
PR#126
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/113#issuecomment-37555992
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10576181
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -149,14 +151,9 @@ private[spark] class MapOutputTracker(conf: SparkConf
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10578300
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
@@ -17,28 +17,24 @@
package org.apache.spark.scheduler
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10579167
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -181,15 +178,50 @@ private[spark] class MapOutputTracker(conf:
SparkConf
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10580179
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10580856
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10584092
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10585234
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
@@ -17,28 +17,24 @@
package org.apache.spark.scheduler
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10540258
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -50,23 +54,26 @@ private[spark] class
MapOutputTrackerMasterActor(tracker
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10540325
--- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10540432
--- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10540569
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -181,15 +186,49 @@ private[spark] class MapOutputTracker(conf:
SparkConf
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10543160
--- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10543349
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -181,15 +186,49 @@ private[spark] class MapOutputTracker(conf:
SparkConf
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/126#issuecomment-37484105
It's true that finalizers are not predictable, but what this patch is doing
is no worse than before. Cleanup() logic is decoupled from finalize(); it can
still
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/126#issuecomment-37484584
This is not specifically related to your patch, but I think we can remove
the MetadataCleaner in SparkContext that cleans up persisted RDDs periodically.
In particular
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10549481
--- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10551095
--- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504353
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -229,10 +243,60 @@ object SparkEnv extends Logging {
broadcastManager
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504474
--- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala
---
@@ -17,13 +17,17 @@
package org.apache.spark.storage
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37372997
@pwendell and @kayousterhout - Thanks for taking the first pass through
this patch. I have made the relevant changes we discussed and updated to master
(non-trivial
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10455635
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10457836
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10458118
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10461463
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/93#discussion_r10362394
--- Diff: python/pyspark/rdd.py ---
@@ -628,6 +669,26 @@ def mergeMaps(m1, m2):
m1[k] += v
return m1
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10363030
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -625,6 +653,30 @@ private[spark] class Master(host: String, port: Int
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10368525
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10368558
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -19,33 +19,43 @@ package org.apache.spark.scheduler
import
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10368594
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -164,9 +167,18 @@ object SparkEnv extends Logging
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36821926
Looks like test failed because of git problems. Retesting later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/50#discussion_r10245880
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -71,10 +71,21 @@ private[spark] class CacheManager(blockManager:
BlockManager
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10283514
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10284454
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10284630
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/44#discussion_r10237674
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -847,6 +847,8 @@ class SparkContext(
partitions: Seq[Int
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10240683
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10183037
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -339,8 +217,8 @@ class JobLogger(val user: String, val logDirName:
String
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10187737
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
52 matches
Mail list logo
