[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-09-12 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/577#discussion_r17482426 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -73,7 +73,17 @@ private[spark] class DiskStore(blockManager: BlockManager

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-09-12 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/577#discussion_r17482448 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -73,7 +73,17 @@ private[spark] class DiskStore(blockManager: BlockManager

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-09-13 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/577#issuecomment-55498864 Don't know if Windows follows the same semantics, though. Never tested it. But I remember that if some file is using, Windows will prevent from deleting

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-09-16 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/577#issuecomment-55755661 @andrewor14 any further suggestion? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-09-16 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/577#discussion_r17642519 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -73,7 +73,21 @@ private[spark] class DiskStore(blockManager: BlockManager

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-09-16 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/577#issuecomment-55840427 Thank for reviewing. Already fixed the minor problems. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-09-17 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/577#discussion_r17704914 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -73,7 +73,21 @@ private[spark] class DiskStore(blockManager: BlockManager

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-09-17 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-55985572 @JoshRosen do you think it's OK? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-09-25 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-56909022 Thank you @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3741] Make ConnectionManager propagate ...

2014-09-30 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2593 [SPARK-3741] Make ConnectionManager propagate errors properly and add mo... ...re logs to avoid Executors swallowing errors This PR made the following changes: * Register a callback

[GitHub] spark pull request: [SPARK-3741] Make ConnectionManager propagate ...

2014-09-30 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2593#issuecomment-57291052 retest this please. This test is OK in my machine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-3777] Display Executor ID for Tasks i...

2014-10-03 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2642 [SPARK-3777] Display Executor ID for Tasks in Stage page Now the Stage page only displays Executor(host) for tasks. However, there may be more than one Executors running in the same host. Currently

[GitHub] spark pull request: [SPARK-3777] Display Executor ID for Tasks i...

2014-10-04 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2642#issuecomment-57909316 As horizontal space is precious for including more metrics, might it make sense to combine Address / Executor and Executor ID into a single Executor column, with values

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-10-05 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/577#discussion_r18433883 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -73,7 +73,21 @@ private[spark] class DiskStore(blockManager: BlockManager

[GitHub] spark pull request: SPARK-2387: remove stage barrier

2014-07-22 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/1328#discussion_r15220164 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -340,6 +459,7 @@ private[spark] class MapOutputTrackerMaster(conf: SparkConf

[GitHub] spark pull request: SPARK-2387: remove stage barrier

2014-07-22 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/1328#discussion_r15266906 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -340,6 +459,7 @@ private[spark] class MapOutputTrackerMaster(conf: SparkConf

[GitHub] spark pull request: SPARK-2387: remove stage barrier

2014-07-22 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/1328#discussion_r15268635 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -340,6 +459,7 @@ private[spark] class MapOutputTrackerMaster(conf: SparkConf

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-07-22 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/1541 SPARK-2634: Change MapOutputTrackerWorker.mapStatuses to ConcurrentHashMap MapOutputTrackerWorker.mapStatuses is used concurrently, it should be thread-safe. This bug has already been fixed

[GitHub] spark pull request: SPARK-2638 MapOutputTracker concurrency improv...

2014-07-23 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/1542#discussion_r15283935 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -130,7 +130,7 @@ private[spark] abstract class MapOutputTracker(conf: SparkConf

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-07-23 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-49865627 Instead of a ConcurrentHashMap, we should actually move it to a disk backed Map Agree. Is there a PR ready? I think this is a critical bug and hope it can

[GitHub] spark pull request: SPARK-2638 MapOutputTracker concurrency improv...

2014-07-23 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/1542#discussion_r15326142 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -130,7 +130,7 @@ private[spark] abstract class MapOutputTracker(conf: SparkConf

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-07-24 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-50097007 When is this accessed concurrently? For example, HashShuffleReader.read - object BlockStoreShuffleFetcher.fetch - MapOutputTracker.getServerStatuses. Different

[GitHub] spark pull request: SPARK-2638 MapOutputTracker concurrency improv...

2014-07-25 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/1542#discussion_r15392537 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -162,9 +164,9 @@ private[spark] abstract class MapOutputTracker(conf: SparkConf

[GitHub] spark pull request: SPARK-2638 MapOutputTracker concurrency improv...

2014-07-26 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1542#issuecomment-50236563 @javadba, sorry that maybe my previous comment is not clear. I'm opposed to use `synchronized` on `val monitor = shuffleId.toString.intern`. I see you have

[GitHub] spark pull request: SPARK-1656: Fix potential resource leaks

2014-08-06 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/577#issuecomment-51298306 Resolved the conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3741] Make ConnectionManager propagate ...

2014-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/2593#discussion_r18626201 --- Diff: core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala --- @@ -72,14 +83,32 @@ private[nio] class ConnectionManager

[GitHub] spark pull request: [SPARK-3741] Make ConnectionManager propagate ...

2014-10-08 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2593#issuecomment-58461588 Merged and updated the naming. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...

2014-10-09 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2732 [SPARK-3877] Throw an exception when application is not successful so that the exit code wil be set to 1 When an yarn application fails (yarn-cluster mode), the exit code of spark-submit is still

[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...

2014-10-09 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-58522681 I think yarn client mode is another story. Driver can detect the failure of executors by itself since it runs in the local. And if the driver crashes, it means

[GitHub] spark pull request: [SPARK-3887] Send stracktrace in ConnectionMan...

2014-10-10 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2741#issuecomment-58621671 It's really helpful :+1: Just a small question: Should specify charset when converting between bytes and String, or Spark always assumes the environment uses UTF8

[GitHub] spark pull request: [SPARK-3887] Send stracktrace in ConnectionMan...

2014-10-12 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2741#issuecomment-58808996 LGTM. Is it worth to add `val UTF8 = Charset.forName(UTF-8)` into `object Utils`? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [Spark 3922] Refactor spark-core to use Utils....

2014-10-13 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2781 [Spark 3922] Refactor spark-core to use Utils.UTF_8 A global UTF8 constant is very helpful to handle encoding problems when converting between String and bytes. There are several solutions here

[GitHub] spark pull request: [Spark 3922] Refactor spark-core to use Utils....

2014-10-13 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2781#issuecomment-58854034 /cc @rxin, @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [Spark 3922] Refactor spark-core to use Utils....

2014-10-13 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2781#issuecomment-58984965 Also, it might be worth it to import the constant directory and just use UTF_8 (instead of something like Utils.UTF_8) so that switching to the JDK7 version later

[GitHub] spark pull request: Add afterExecute for handleConnectExecutor

2014-10-14 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2794 Add afterExecute for handleConnectExecutor Sorry. I found that I forgot to add `afterExecute` for `handleConnectExecutor` in #2593. You can merge this pull request into a Git repository by running

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-16 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59458388 @vanzin Thanks for reviewing this one. Already updated as per your suggestions. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-16 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59459888 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-17 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59499788 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-17 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59596417 Already updated the docs and the failure message. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-18 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59601473 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4008] Fix kryo with fold in KryoSeria...

2014-10-20 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2856 [SPARK-4008] Fix kryo with fold in KryoSerializerSuite `zeroValue` will be serialized by `spark.closure.serializer` but `spark.closure.serializer` only supports the default Java serializer. So

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-20 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59870049 Could you also take a look at #2748, backport for branch 1.1? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: Fix a wrong format specifier

2014-10-21 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2875 Fix a wrong format specifier Just found a typo. Should not use %f for Long. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK

[GitHub] spark pull request: [SPARK-4035] Fix a wrong format specifier

2014-10-21 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2875#issuecomment-59917911 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-22 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-60078393 @andrewor14 is it OK to merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-24 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2748#issuecomment-60377001 Already merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-24 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/2748 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [Spark 3922] Refactor spark-core to use Utils....

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2781#issuecomment-60556812 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2957 [SPARK-4097] Fix the race condition of 'thread' There is a chance that `thread` is null when calling `thread.interrupt()`. ```Scala override def cancel(): Unit = this.synchronized

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60565342 You can also just take a local reference to the thread, and operate on it. The local reference will of course be null in both cases or not-null in both cases

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60577583 The usage of `run` is in `AsyncRDDActions.takeAsync`. It uses `scala.concurrent.ExecutionContext.Implicits.global` as the implicit `ExecutionContext`. Because

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60579222 How would running another task X change the value of t in T1's stack? No, I mean another task X will observe `t.interrupt()` because task X happens to run

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60580481 No, I checked `ThreadPoolExecutor` here: http://www.grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/concurrent/ThreadPoolExecutor.java

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60580846 In summary, the local variable solution may interrupt the thread during running another new task. So that's a potential problem. Although my solution here may

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60581389 But the call to interrupt the old task interrupt happens after the new task begins on the thread. Because I use the same lock, this should not happen

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60581813 There are only two cases: ```Scala t1: if (thread != null) { thread.interrupt() } t2: thread = null

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60583790 You example assumes `ComplexFutureAction.run` may be called more than once. However after reviewing other codes, I don't think `ComplexFutureAction.run` can be called

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60584045 If `ComplexFutureAction.run` is called only once, I think my current fix is enough. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60584425 Here is the usage of `ComplexFutureAction`: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala#L66 I

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60584794 @srowen thank you for reviewing it:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-28 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/2957#discussion_r19455903 --- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala --- @@ -210,7 +210,9 @@ class ComplexFutureAction[T] extends FutureAction[T

[GitHub] spark pull request: [SPARK-4097] Fix the race condition of 'thread...

2014-10-28 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2957#issuecomment-60857035 @rxin Done. Is it OK? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...

2014-10-29 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/2980#discussion_r19533353 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala --- @@ -56,6 +57,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab

[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...

2014-10-29 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/2980#discussion_r19533499 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala --- @@ -145,6 +148,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab

[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...

2014-10-29 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/2980#discussion_r19535011 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala --- @@ -145,6 +148,7 @@ private[ui] class ExecutorsPage(parent: ExecutorsTab

[GitHub] spark pull request: [SPARK-4153][WebUI] Update the sort keys for H...

2014-10-30 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/3014 [SPARK-4153][WebUI] Update the sort keys for HistoryPage Sort Started, Completed, Duration and Last Updated by time. You can merge this pull request into a Git repository by running: $ git

[GitHub] spark pull request: [SPARK-4163][Core][Web UI] Send the fetch fail...

2014-10-30 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/3032 [SPARK-4163][Core][Web UI] Send the fetch failure message back to Web UI This is a PR to send the fetch failure message back to Web UI. Before: ![f1](https://cloud.githubusercontent.com

[GitHub] spark pull request: Display the executor ID in the Web UI when Exe...

2014-10-31 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/3033 Display the executor ID in the Web UI when ExecutorLostFailure happens Now when ExecutorLostFailure happens, it only displays `ExecutorLostFailure (executor lost)`. Adding the executor id will help

[GitHub] spark pull request: SPARK-1278: Fix improper use of SimpleDateForm...

2014-03-19 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/179 SPARK-1278: Fix improper use of SimpleDateFormat `SimpleDateFormat` is not immutable and thread-safe. Some places use the same SimpleDateFormat object without safeguard in the multiple threads

[GitHub] spark pull request: SPARK-1278: Fix improper use of SimpleDateForm...

2014-03-20 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/179#issuecomment-38141424 Overrode the previous commit and updated to the latest master codes. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: SPARK-1278: Fix improper use of SimpleDateForm...

2014-03-21 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/179#discussion_r10860446 --- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala --- @@ -23,12 +23,15 @@ import java.util.Date /** * Utilities used throughout the web

[GitHub] spark pull request: Merge Hadoop Into Spark

2014-04-01 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/286#issuecomment-39284253 You know, ever since those .orbit things were added, sbt update costs nearly 40 minutes to resolve all the dependencies in mainland China... I have to turn it off

[GitHub] spark pull request: SPARK-1382: Fix NPE in DStream.slice

2014-04-08 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/365 SPARK-1382: Fix NPE in DStream.slice If we call the DStream.slice() before StreamingContext.start() has been called, then zeroTime is still null, and it will throw a null pointer exception

[GitHub] spark pull request: SPARK-1482: Fix potential resource leaks in sa...

2014-04-13 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/400 SPARK-1482: Fix potential resource leaks in saveAsHadoopDataset and save... ...AsNewAPIHadoopDataset JIRA: https://issues.apache.org/jira/browse/SPARK-1482 You can merge this pull request

[GitHub] spark pull request: SPARK-1382: Fix NPE in DStream.slice

2014-04-16 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/365#discussion_r11716967 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala --- @@ -725,6 +725,9 @@ abstract class DStream[T: ClassTag

[GitHub] spark pull request: SPARK-1382: Fix NPE in DStream.slice

2014-04-17 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/365#issuecomment-40685083 Changing to `SparkExcepation` should not break the API since `SparkExcepation` is a subclass of `Exception`. --- If your project is set up for it, you can reply

[GitHub] spark pull request: SPARK-1583: Fix a bug that using java.util.Has...

2014-04-23 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/500 SPARK-1583: Fix a bug that using java.util.HashMap by mistake JIRA: https://issues.apache.org/jira/browse/SPARK-1583 Does anyone know why using `java.util.HashMap` rather than

[GitHub] spark pull request: SPARK-1583: Fix a bug that using java.util.Has...

2014-04-23 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/500#issuecomment-41135796 If I understand correctly, `SaveStageAndTaskInfo` in `SparkListenerSuite` lacks appropriate synchronization to guarantee memory visibility. That's why this test fails

[GitHub] spark pull request: SPARK-1589: Fix the incorrect compare

2014-04-23 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/508 SPARK-1589: Fix the incorrect compare JIRA: https://issues.apache.org/jira/browse/SPARK-1589 You can merge this pull request into a Git repository by running: $ git pull https://github.com

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-09-01 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-54067191 ping @JoshRosen, could you help take a look at this one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-09-01 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-54103100 the output status for a particular map task should be immutable once written. But `mapStatuses` is mutable. Therefore, some thread is putting an item

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-09-01 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-54103752 You remind me that even if `Array[MapStatus]` won't be modified, according to java memory model, fetchedStatuses (https://github.com/zsxwing/spark/blob/SPARK-2634/core

[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...

2014-09-01 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-54104158 ConcurrentHashMap should fix all these issues I found. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-4166][Core][WebUI] Display the executor...

2014-10-31 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3033#issuecomment-61357949 I updated to use `Utils.jsonOption` to support backward compatible. Also update #3032 --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-01 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3032#issuecomment-61373259 @aarondav thanks for reviewing it. Updated as per your comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-01 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19705166 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -313,14 +319,30 @@ object ShuffleBlockFetcherIterator

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-01 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19709864 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala --- @@ -52,18 +53,18 @@ private[hash] object

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-01 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19709867 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -313,14 +319,29 @@ object ShuffleBlockFetcherIterator

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-01 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19709873 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala --- @@ -52,18 +53,18 @@ private[hash] object

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-02 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3032#issuecomment-61396504 How about now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-02 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19710088 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -313,14 +319,30 @@ object ShuffleBlockFetcherIterator

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-02 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19710310 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala --- @@ -52,18 +53,18 @@ private[hash] object

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-02 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19718354 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala --- @@ -52,21 +53,21 @@ private[hash] object

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-02 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19718379 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1060,7 +1060,7 @@ class DAGScheduler

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-02 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3032#issuecomment-61441682 New screenshots for current codes. Stages page: ![stags](https://cloud.githubusercontent.com/assets/1000778/4878803/e2ffede6-6318-11e4-8376-15854e2f6eb3.png

[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...

2014-11-02 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3032#discussion_r19719301 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala --- @@ -52,21 +53,21 @@ private[hash] object

[GitHub] spark pull request: [SPARK-4204][Core][WebUI] Change Utils.excepti...

2014-11-03 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/3073 [SPARK-4204][Core][WebUI] Change Utils.exceptionString to contain the inner exceptions and make the error information in Web UI more friendly This PR fixed `Utils.exceptionString` to output

[GitHub] spark pull request: [SPARK-4166][Core] Add a backward compatibilit...

2014-11-03 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3085#issuecomment-61581027 /cc @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4166][Core] Add a backward compatibilit...

2014-11-03 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/3085 [SPARK-4166][Core] Add a backward compatibility test for ExecutorLostFailure You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark

[GitHub] spark pull request: [SPARK-4166][Core][WebUI] Display the executor...

2014-11-03 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3033#issuecomment-61581056 Already added it in #3085 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

  1   2   3   4   5   6   7   8   9   10   >