[GitHub] spark pull request: SPARK-2098: All Spark processes should support...

2014-09-11 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-55353905 OK, I generally understand what you mean,I will re-implement the feature at the weekend. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-12 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55391092 @mengxr @allwefantasy The current broadcast-based implementation, especially in the corpus is large, the performance loss is more serious. Next week I

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1983#discussion_r17490277 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala --- @@ -0,0 +1,397 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-3517]mapPartitions is not correct clear...

2014-09-12 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2376 [SPARK-3517]mapPartitions is not correct clearing up the closure You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-3517

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-12 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55479363 @allwefantasy I have updated the code, you can try the latest code. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-2947] DAGScheduler resubmit the stage i...

2014-09-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1877#discussion_r17510561 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1046,41 +1046,37 @@ class DAGScheduler( case FetchFailed

[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...

2014-09-12 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2376#issuecomment-55482720 @rxin Code like this: ``` val topicModel = Big object val broadcastModel = data.context.broadcast(topicModel) corpus

[GitHub] spark pull request: [SPARK-1477]: Add the lifecycle interface

2014-09-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/991#discussion_r17510639 --- Diff: core/src/main/scala/org/apache/spark/Service.scala --- @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-13 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2379 [SPARK-2098] All Spark processes should support spark-defaults.conf, config file This is another implementation about #1256 cc @andrewor14 @vanzin You can merge this pull request into a Git

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-13 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-55485064 OK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-13 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/1256 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]LDA based on Graphx

2014-09-14 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2388 [WIP][SPARK-1405][MLLIB]LDA based on Graphx cc @mengxr You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark graphx_lda Alternatively you

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]LDA based on Graphx

2014-09-15 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-55617366 简要的说明: * 图的结构 顶点为词(the source vertex),文档(the target vertex).边为文档中的词对应的主题(ID数ç

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-16 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r17641601 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala --- @@ -47,14 +48,15 @@ private[spark] class WorkerArguments(args: Array

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-16 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2379#issuecomment-55838336 @andrewor14 @vanzin This is a very good idea, we create a SparkConf from the beginning and we will use that for the entire process. But it need to modify a lot of code

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-16 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r17644167 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala --- @@ -44,30 +50,19 @@ private[spark] class HistoryServerArguments

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-16 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r17646251 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala --- @@ -47,14 +48,15 @@ private[spark] class WorkerArguments(args: Array

[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...

2014-09-17 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2376#issuecomment-55913971 I tried to reproduce it with the way the test case, but was unsuccessful.But like the code below does appear the problem, do not know why ```scala def

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-17 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2379#issuecomment-55982531 OK, the code has been updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2815]: Compilation failed upon the hado...

2014-09-18 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1754#issuecomment-55998415 @pwendell I have updated the regular expression. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1477]: Add the lifecycle interface

2014-09-18 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/991 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-18 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r17744702 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala --- @@ -47,14 +48,25 @@ private[spark] class WorkerArguments(args: Array

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-18 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r17766151 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala --- @@ -44,30 +50,19 @@ private[spark] class HistoryServerArguments

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-19 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2379#issuecomment-56253206 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-19 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2379#issuecomment-56253720 Don't know why no trigger Jenkins retest thi,the code should be no problem. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: SPARK-3625'In some cases, the RDD.checkpoint d...

2014-09-21 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2480 SPARK-3625'In some cases, the RDD.checkpoint does not work You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-3625 Alternatively

[GitHub] spark pull request: [SPARK-3293] yarn's web show SUCCEEDED when ...

2014-09-23 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-56525297 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-24 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r18011958 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1357,6 +1357,58 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-25 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2379#issuecomment-56810769 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-25 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r18070980 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1357,6 +1357,58 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request: [SPARK-3293] yarn's web show SUCCEEDED when ...

2014-09-25 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-56912003 Ok, no problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3293] yarn's web show SUCCEEDED when ...

2014-09-25 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-56914209 Yes, this does not involve `org.apache.spark.deploy.yarn.Client` class, which run outside the cluster. We should call `YarnClient.killApplication` when an uncaught

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]collapsed Gibbs sampli...

2014-09-26 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/1983 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-26 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r18121155 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1357,6 +1357,58 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-26 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2379#discussion_r18121404 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1357,6 +1357,58 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request: [SPARK-2098] All Spark processes should suppor...

2014-09-26 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2379#issuecomment-57040105 @andrewor14 The code has been updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...

2014-09-28 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/2376 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [WIP][SPARK-3517]mapPartitions is not correct ...

2014-09-28 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2376#issuecomment-57078975 I temporarily can not reproduce it, and close this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] make i...

2014-09-28 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2524#discussion_r18128796 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -112,6 +112,10 @@ class DAGScheduler( // stray messages

[GitHub] spark pull request: [SPARK-1720][SPARK-1719] Add the value of LD_L...

2014-09-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-57199913 The current solution is simple to implement.If we don't do so,we have to change a lot of existing code. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-3627] - [yarn] - fix exit code and fina...

2014-09-29 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/2577#discussion_r18170801 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -450,6 +539,15 @@ object ApplicationMaster extends Logging

[GitHub] spark pull request: [SPARK-3293] yarn's web show SUCCEEDED when ...

2014-09-29 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/2311 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-1720][SPARK-1719] Add the value of LD_L...

2014-10-01 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-57572266 Ok, I'll try to use LD_LIBRARY_PATH. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [WIP][SPARK-3625][SPARK-3623]Graph should supp...

2014-10-02 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2631 [WIP][SPARK-3625][SPARK-3623]Graph should support the checkpoint operation You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-3623

[GitHub] spark pull request: Fix NPE for JsonProtocol

2014-07-21 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/1511 Fix NPE for JsonProtocol You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark JsonProtocol Alternatively you can review and apply

[GitHub] spark pull request: [WIP]Spark 2037: yarn client mode doesn't supp...

2014-07-22 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1180#issuecomment-49828369 @tgravescs Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2644] Hive should not be enabled by def...

2014-07-23 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1546#issuecomment-49852813 [HiveFromSpark](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/sql/hive/HiveFromSpark.scala#L22) class is dependent

[GitHub] spark pull request: [WIP]Spark 2037: yarn client mode doesn't supp...

2014-07-23 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1180#issuecomment-49874821 If we do not add this, When spark has failed in the yarn.SparkContext's progress will be hang. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: Spark 2037: yarn client mode doesn't support s...

2014-07-23 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1180#issuecomment-49881861 A little error repair at once. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: Spark 2037: yarn client mode doesn't support s...

2014-07-23 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1180#issuecomment-49886617 Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: SPARK-2662: Fix NPE for JsonProtocol

2014-07-23 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1511#issuecomment-49959000 @mateiz Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: Build should not run hive tests by default.

2014-07-23 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/1565 Build should not run hive tests by default. cc @pwendell @ScrapCodes You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-2484

[GitHub] spark pull request: Localise hadoop configuration when submitting ...

2014-07-24 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1574#issuecomment-50024391 This is a good idea. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2037]: yarn client mode doesn't support...

2014-07-24 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1180#discussion_r15352693 --- Diff: yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala --- @@ -115,7 +117,33 @@ private[spark] class

[GitHub] spark pull request: [SPARK-2037]: yarn client mode doesn't support...

2014-07-24 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1180#discussion_r15352756 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala --- @@ -282,15 +298,22 @@ class ExecutorLauncher(args

[GitHub] spark pull request: [SPARK-2037]: yarn client mode doesn't support...

2014-07-24 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1180#issuecomment-50044511 This situation exists in theory. [SparkContext.scala#L992](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L992) deal

[GitHub] spark pull request: [SPARK-2670] FetchFailedException should be th...

2014-07-26 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1578#issuecomment-50232324 Here also should throw an `FetchFailedException `? ```scala override def next(): (BlockId, Option[Iterator[Any]]) = { resultsGotten += 1 val

[GitHub] spark pull request: ConnectionManager throws out of Could not fin...

2014-07-26 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/1603 ConnectionManager throws out of Could not find reference for received ack message xxx exception. You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] spark pull request: Build should not run hive tests by default.

2014-07-26 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1565#issuecomment-50237538 @ScrapCodes I think that the solution is simple and effective, is a better. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: ConnectionManager throws out of Could not fin...

2014-07-26 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1603#issuecomment-50239511 Throw an exception here cause `System.exit(ExecutorExitCode.UNCAUGHT_EXCEPTION)` is called. This is not necessary. --- If your project is set up for it, you can reply

[GitHub] spark pull request: Spark-2447 : Spark on HBase

2014-07-26 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1608#discussion_r15437166 --- Diff: external/hbase/pom.xml --- @@ -0,0 +1,217 @@ +project xmlns=http://maven.apache.org/POM/4.0.0; xmlns:xsi=http://www.w3.org/2001/XMLSchema

[GitHub] spark pull request: Spark-2447 : Spark on HBase

2014-07-26 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1608#discussion_r15437173 --- Diff: external/hbase/pom.xml --- @@ -0,0 +1,217 @@ +project xmlns=http://maven.apache.org/POM/4.0.0; xmlns:xsi=http://www.w3.org/2001/XMLSchema

[GitHub] spark pull request: Spark-2447 : Spark on HBase

2014-07-26 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1608#discussion_r15437185 --- Diff: external/hbase/pom.xml --- @@ -0,0 +1,217 @@ +project xmlns=http://maven.apache.org/POM/4.0.0; xmlns:xsi=http://www.w3.org/2001/XMLSchema

[GitHub] spark pull request: [SPARK-1997] update breeze to version 0.8.1

2014-07-27 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/940#issuecomment-50293501 @mengxr Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-1997] update breeze to version 0.8.1

2014-07-27 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/940#issuecomment-50297230 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15446910 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15446921 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -176,10 +406,17 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447384 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447402 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447397 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447401 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447436 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-27 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447430 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-28 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447449 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-28 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447510 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -935,15 +941,22 @@ private[spark] object Utils extends Logging { * Currently

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-28 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447541 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala --- @@ -353,26 +368,53 @@ class ExternalAppendOnlyMap[K, V, C

[GitHub] spark pull request: [SPARK-2532] WIP Consolidated shuffle fixes

2014-07-28 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1609#discussion_r15447535 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala --- @@ -353,26 +368,53 @@ class ExternalAppendOnlyMap[K, V, C

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-28 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/1619 [WIP][SPARK-2677]BasicBlockFetchIterator#next can wait forever You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-2677

[GitHub] spark pull request: [SPARK-2677] BasicBlockFetchIterator#next can ...

2014-07-29 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1632#discussion_r15508224 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -117,31 +121,45 @@ object BlockFetcherIterator

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1619#issuecomment-50456598 @sarutak I think add a heartbeat detection mechanism is a good solution --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1619#issuecomment-50458991 @sarutak [ConnectionManager.scala#L259](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/network/ConnectionManager.scala#L259) to deal

[GitHub] spark pull request: [SPARK-2544][MLLIB] Improve ALS algorithm reso...

2014-07-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-50492147 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2544][MLLIB] Improve ALS algorithm reso...

2014-07-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-50503214 @mengxr Done. Tomorrow, I will test in detail. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2199] [mllib] topic modeling

2014-07-29 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1269#discussion_r15560968 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/clustering/topicmodeling/topicmodels/RobustPLSASuite.scala --- @@ -0,0 +1,40 @@ +/* + * Licensed

[GitHub] spark pull request: [SPARK-2544][MLLIB] Improve ALS algorithm reso...

2014-07-29 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/929#discussion_r15566577 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala --- @@ -255,6 +255,9 @@ class ALS private ( rank, lambda, alpha

[GitHub] spark pull request: [SPARK-2544][MLLIB] Improve ALS algorithm reso...

2014-07-29 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/929#discussion_r15566754 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala --- @@ -255,6 +255,9 @@ class ALS private ( rank, lambda, alpha

[GitHub] spark pull request: [SPARK-2544][MLLIB] Improve ALS algorithm reso...

2014-07-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-50574926 Ok, I will try it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2098: All Spark processes should support...

2014-07-30 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-50586734 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50706018 We can also merge it to 1.0 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-2766: ScalaReflectionSuite throw an lleg...

2014-07-31 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/1683 SPARK-2766: ScalaReflectionSuite throw an llegalArgumentException in JDK 6 You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK

[GitHub] spark pull request: SPARK-2098: All Spark processes should support...

2014-07-31 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-50718557 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2766: ScalaReflectionSuite throw an lleg...

2014-07-31 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1683#issuecomment-50841416 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1470][SPARK-1842] Use the scala-logging...

2014-08-01 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1369#issuecomment-50851936 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1997] update breeze to version 0.8.1

2014-08-01 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/940#issuecomment-50855187 @mengxr done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-1997] update breeze to version 0.8.1

2014-08-01 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/940#discussion_r15684437 --- Diff: mllib/pom.xml --- @@ -60,6 +60,14 @@ groupIdjunit/groupId artifactIdjunit/artifactId /exclusion

[GitHub] spark pull request: [SPARK-1997] update breeze to version 0.8.1

2014-08-01 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/940#issuecomment-50860276 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1997] update breeze to version 0.8.1

2014-08-01 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/940#issuecomment-50906124 How do we resolve this issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1470][SPARK-1842] Use the scala-logging...

2014-08-01 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1369#issuecomment-50910604 @srowen Thank you for your comments. And how to modify `GenerateMIMAIgnore` file to exclude a method? An example? --- If your project is set up for it, you can reply

[GitHub] spark pull request: SPARK-2800: Exclude scalastyle-output.xml Apac...

2014-08-01 Thread witgo
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/1729 SPARK-2800: Exclude scalastyle-output.xml Apache RAT checks You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-2800 Alternatively

[GitHub] spark pull request: [SPARK-1470][SPARK-1842] Use the scala-logging...

2014-08-01 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1369#issuecomment-50951603 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

  1   2   3   4   5   6   7   8   9   >