[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3891#issuecomment-68833142 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala(override the...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3898#issuecomment-68833371 Finally, please file a JIRA and add it to the PR title. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3820#issuecomment-68834477 Thanks for doing this, I've been getting a ton of requests for this feature! Can you also add this to the sql programming guide? --- If your project is set up

[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...

2015-01-05 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3696#issuecomment-68834508 LGTM, so I'm going to merge this into `master` (1.3.0). Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...

2015-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3696 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3891#issuecomment-68832961 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3891#issuecomment-68833139 [Test build #25090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25090/consoleFull) for PR 3891 at commit

[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3891#issuecomment-68833133 [Test build #25090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25090/consoleFull) for PR 3891 at commit

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala(override the...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3898#issuecomment-68833152 Can you also add a regression test to

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala(override the...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3898#issuecomment-68833170 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala(override the...

2015-01-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/3898#discussion_r22511002 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -427,6 +427,13 @@ private[hive] case class MetastoreRelation

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala(override the...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3898#issuecomment-68833443 [Test build #25091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25091/consoleFull) for PR 3898 at commit

[GitHub] spark pull request: SPARK-4226: Add support for subqueries in wher...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3888#issuecomment-68833871 This is simpler, but it has several disadvantages to the other approach: - The InSet it collected to the driver and thus could cause OOMs when large - I don't

[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...

2015-01-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/3820#discussion_r22511350 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala --- @@ -141,6 +142,12 @@ private[sql] trait SQLConf {

[GitHub] spark pull request: [SPARK-2165][YARN]add support for setting maxA...

2015-01-05 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3878#issuecomment-68681027 My opinion is that this is more of a general app property than an AM property, so I'd go for `spark.yarn.maxAppAttempts` instead. That also avoids confusion with the fact

[GitHub] spark pull request: [Minor][Mllib] Simplify loss function

2015-01-05 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/3899 [Minor][Mllib] Simplify loss function This is a minor pr where I think that we can simply take minus of `margin` here, instead of subtracting `margin`. Mathematically, they are equal. But

[GitHub] spark pull request: [Minor][Mllib] Simplify loss function

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3899#issuecomment-68686539 [Test build #25053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25053/consoleFull) for PR 3899 at commit

[GitHub] spark pull request: [SPARK-4631] unit test for MQTT

2015-01-05 Thread Bilna
Github user Bilna commented on the pull request: https://github.com/apache/spark/pull/3844#issuecomment-68686800 @tdas, Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [Minor][Mllib] Simplify loss function

2015-01-05 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3899#issuecomment-68687443 +1 looks like a small good improvement. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-4631] unit test for MQTT

2015-01-05 Thread Bilna
Github user Bilna commented on a diff in the pull request: https://github.com/apache/spark/pull/3844#discussion_r22453522 --- Diff: external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala --- @@ -17,31 +17,111 @@ package

[GitHub] spark pull request: [SPARK-1600] Refactor FileInputStream tests to...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3801#issuecomment-68688402 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-1600] Refactor FileInputStream tests to...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3801#issuecomment-68688395 [Test build #25052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25052/consoleFull) for PR 3801 at commit

[GitHub] spark pull request: [Minor][Mllib] Simplify loss function

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3899#issuecomment-68693770 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-2165][YARN]add support for setting maxA...

2015-01-05 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3878#issuecomment-68693775 @sryza Thanks. That makes sense. @tgravescs How do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [Minor][Mllib] Simplify loss function

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3899#issuecomment-68693757 [Test build #25053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25053/consoleFull) for PR 3899 at commit

[GitHub] spark pull request: [STREAMING] SPARK-4986 Wait for receivers to d...

2015-01-05 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/3868#issuecomment-68689798 I am happy to review the code if you take a pass on implementing (2). I can jump in if things get too hairy. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-68690112 [Test build #25054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25054/consoleFull) for PR 3222 at commit

[GitHub] spark pull request: [SPARK-4504] Minor bug fixes in bin/run-exampl...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3069#issuecomment-68692219 [Test build #25055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25055/consoleFull) for PR 3069 at commit

[GitHub] spark pull request: [SPARK-4908][SQL] Prevent multiple concurrent ...

2015-01-05 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3834#issuecomment-68692301 Just for reference, the root cause behind this issue is discussed in [SPARK-4908] [1]. [1]:

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala

2015-01-05 Thread seayi
Github user seayi commented on the pull request: https://github.com/apache/spark/pull/3898#issuecomment-68683132 override the sameresult method only compare databasename and table name --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala

2015-01-05 Thread seayi
GitHub user seayi opened a pull request: https://github.com/apache/spark/pull/3898 Update HiveMetastoreCatalog.scala modify the sameresult method only compare databasename and table name because in previous : cache table t1; select count(*) from t1; it will

[GitHub] spark pull request: [STREAMING] SPARK-4986 Wait for receivers to d...

2015-01-05 Thread cleaton
Github user cleaton commented on the pull request: https://github.com/apache/spark/pull/3868#issuecomment-68678753 @tdas Thank you for the input. Yes, the main purpose of this patch is to make ReceiverTracker graceful by waiting for ssc.sparkContext.runJob(tempRDD,

[GitHub] spark pull request: [SPARK-5088] Use spark-class for running execu...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3897#issuecomment-68680860 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5088] Use spark-class for running execu...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3897#issuecomment-68680857 [Test build #25051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25051/consoleFull) for PR 3897 at commit

[GitHub] spark pull request: [SPARK-1600] Refactor FileInputStream tests to...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3801#issuecomment-68683294 [Test build #25052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25052/consoleFull) for PR 3801 at commit

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3898#issuecomment-68683258 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Update HiveMetastoreCatalog.scala

2015-01-05 Thread seayi
Github user seayi commented on the pull request: https://github.com/apache/spark/pull/3898#issuecomment-68683408 i test with hive table,after modify the sameresult method it is ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-1600] Refactor FileInputStream tests to...

2015-01-05 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3801#issuecomment-68684233 Pushed some commits addressing most of the feedback, but I'm still struggling to remove that last `Thread.sleep(1000)`. I think that the problem here is that the

[GitHub] spark pull request: [SPARK-4991][CORE] Worker should reconnect to ...

2015-01-05 Thread liyezhang556520
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/3825#issuecomment-68684212 @JoshRosen , If we want to use the supervision mechanism. We need to add another actor level as parent of the current Master actor. I don't know if that is

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-68696993 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-68696985 [Test build #25054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25054/consoleFull) for PR 3222 at commit

[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...

2015-01-05 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68698508 This one always confuses me, but here's what I think I know: The compiled `Optional` in Spark won't have the correct (meaning, matching the Google Guava

[GitHub] spark pull request: [SPARK-4504] Minor bug fixes in bin/run-exampl...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3069#issuecomment-68698622 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4504] Minor bug fixes in bin/run-exampl...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3069#issuecomment-68698616 [Test build #25055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25055/consoleFull) for PR 3069 at commit

[GitHub] spark pull request: [SPARK-5036][Graphx]Better support sending par...

2015-01-05 Thread shijinkui
Github user shijinkui commented on the pull request: https://github.com/apache/spark/pull/3866#issuecomment-68710366 @ankurdave @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-5073] spark.storage.memoryMapThreshold ...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3900#issuecomment-68711124 [Test build #25056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25056/consoleFull) for PR 3900 at commit

[GitHub] spark pull request: [SPARK-5073] spark.storage.memoryMapThreshold ...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3900#issuecomment-68711139 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4764] Ensure that files are fetched ato...

2015-01-05 Thread preaudc
Github user preaudc commented on the pull request: https://github.com/apache/spark/pull/2855#issuecomment-68702554 Yes, my bad, {{targetDir}} is indeed already a {{File}}. @JoshRosen , how could I fix this, should I create a new pull request, or can this one be reopened? --- If

[GitHub] spark pull request: [SPARK-5073] spark.storage.memoryMapThreshold ...

2015-01-05 Thread Lewuathe
GitHub user Lewuathe opened a pull request: https://github.com/apache/spark/pull/3900 [SPARK-5073] spark.storage.memoryMapThreshold have two default value Because major OS page sizes is about 4KB, the default value of spark.storage.memoryMapThreshold is integrated to 2 * 4096 You

[GitHub] spark pull request: [SPARK-4631] unit test for MQTT

2015-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/3844#discussion_r22459212 --- Diff: external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala --- @@ -17,31 +17,111 @@ package

[GitHub] spark pull request: [SPARK-5073] spark.storage.memoryMapThreshold ...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3900#issuecomment-68703864 [Test build #25056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25056/consoleFull) for PR 3900 at commit

[GitHub] spark pull request: [SPARK-4504][Examples] fix run-example failure...

2015-01-05 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3377#issuecomment-68706124 There was already a PR for this: https://github.com/apache/spark/pull/3069 But it seems to be fixing a different root cause, that the assembly generated by SBT and

[GitHub] spark pull request: [SPARK-4504] Minor bug fixes in bin/run-exampl...

2015-01-05 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3069#issuecomment-68706026 I think this has been superseded by the discussion in https://github.com/apache/spark/pull/3377 ? --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [STREAMING] SPARK-4986 Wait for receivers to d...

2015-01-05 Thread cleaton
Github user cleaton commented on the pull request: https://github.com/apache/spark/pull/3868#issuecomment-68707568 OK sounds great. :+1: I can prepare an implementation of (2). Bit busy now, but I think I can have something to review in a week. Any specific unit test

[GitHub] spark pull request: [SPARK-4644][Core] Implement skewed join

2015-01-05 Thread lianhuiwang
Github user lianhuiwang commented on a diff in the pull request: https://github.com/apache/spark/pull/3505#discussion_r22460587 --- Diff: core/src/main/scala/org/apache/spark/rdd/SkewedJoinRDD.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2015-01-05 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-68733624 @andrewor14 What to do now? @vanzin @sryza @tgravescs Someone has any better idea? --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-5006][Deploy]spark.port.maxRetries does...

2015-01-05 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3841#issuecomment-68734238 @andrewor14 Could you take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5057]Add more details in log when using...

2015-01-05 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3875#issuecomment-68734038 @JoshRosen Then is it ok? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2309][MLlib] Generalize the binary logi...

2015-01-05 Thread avulanov
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/1379#issuecomment-68741897 @dbtsai Just back from vacation too:) I used my old implementation of the matrix form of back propagation and made sure that it properly uses stride of

[GitHub] spark pull request: [SPARK-4737] Task set manager properly handles...

2015-01-05 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3638#discussion_r22491893 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -256,15 +256,21 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-2429] [MLlib] Hierarchical Implementati...

2015-01-05 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2906#issuecomment-68787775 @yu-iskw @rnowling, I asked @freeman-lab to make one pass on this PR. Let's ping him :) --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-4737] Task set manager properly handles...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3638#issuecomment-68787801 [Test build #25064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25064/consoleFull) for PR 3638 at commit

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-68787824 This looks like a legitimate test failure. Ther AMPLab webserver is having some issues today, so here's a different link to reach the same test result:

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread ksakellis
Github user ksakellis commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-68788875 Yes it is. Not sure why I changed the #of cores between the two commits in the unit test - weird. Anyways. it has been fixed. --- If your project is set up for it,

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-68789356 [Test build #25065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25065/consoleFull) for PR 3711 at commit

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3711#discussion_r22495998 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorDetails.scala --- @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-68798860 [Test build #25065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25065/consoleFull) for PR 3711 at commit

[GitHub] spark pull request: [SPARK-5073] spark.storage.memoryMapThreshold ...

2015-01-05 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/3900#issuecomment-68801129 Agree with 2MB with the caveat that this could cause some slowdown for the other code path (reading cache blocks from disk). However, memory mapping frequently can be

[GitHub] spark pull request: [SPARK-1600] Refactor FileInputStream tests to...

2015-01-05 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3801#discussion_r22491068 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala --- @@ -319,102 +318,141 @@ class CheckpointSuite extends TestSuiteBase

[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...

2015-01-05 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68786110 Latest version LGTM. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...

2015-01-05 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68789982 Although further creep of the unshading-of-the-shading feels risky, it seems to resolve a problem, and is in principle OK on the same grounds that unshading `Optional`

[GitHub] spark pull request: [SPARK-4969][STREAMING][PYTHON] Add binaryReco...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3803#issuecomment-68795630 [Test build #25066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25066/consoleFull) for PR 3803 at commit

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-68796368 I had some minor comments around naming, but overall this looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-4737] Task set manager properly handles...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3638#issuecomment-68797550 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4737] Task set manager properly handles...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3638#issuecomment-68797542 [Test build #25064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25064/consoleFull) for PR 3638 at commit

[GitHub] spark pull request: SPARK-4921. TaskSetManager.dequeueTask returns...

2015-01-05 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3816#issuecomment-68798497 My conclusion (sorry if it was unclear above) was that dequeueTask returning NO_PREF instead of PROCESS_LOCAL should have no effect at all. I still think it's worth

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-68798867 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-5040][SQL] Support expressing unresolve...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3862#issuecomment-68801752 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-2429] [MLlib] Hierarchical Implementati...

2015-01-05 Thread freeman-lab
Github user freeman-lab commented on the pull request: https://github.com/apache/spark/pull/2906#issuecomment-68794407 Hey all, thanks for the nudge =) I've been going through it, will get you feedback ASAP. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3711#discussion_r22495242 --- Diff: core/src/main/java/org/apache/spark/SparkListenerAdapter.java --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-5093] Set spark.network.timeout to 120s...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3903#issuecomment-68794632 [Test build #25063 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25063/consoleFull) for PR 3903 at commit

[GitHub] spark pull request: [SPARK-4969][STREAMING][PYTHON] Add binaryReco...

2015-01-05 Thread freeman-lab
Github user freeman-lab commented on a diff in the pull request: https://github.com/apache/spark/pull/3803#discussion_r22495402 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala --- @@ -233,6 +236,47 @@ class InputStreamsSuite extends

[GitHub] spark pull request: [SPARK-5093] Set spark.network.timeout to 120s...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3903#issuecomment-68794641 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4286] Integrate external shuffle servic...

2015-01-05 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/3861#issuecomment-68795001 @aarondav --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-4969][STREAMING][PYTHON] Add binaryReco...

2015-01-05 Thread freeman-lab
Github user freeman-lab commented on a diff in the pull request: https://github.com/apache/spark/pull/3803#discussion_r22495425 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala --- @@ -233,6 +236,47 @@ class InputStreamsSuite extends

[GitHub] spark pull request: [SPARK-4969][STREAMING][PYTHON] Add binaryReco...

2015-01-05 Thread freeman-lab
Github user freeman-lab commented on a diff in the pull request: https://github.com/apache/spark/pull/3803#discussion_r22496437 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala --- @@ -373,6 +393,25 @@ class StreamingContext private[streaming] (

[GitHub] spark pull request: [SPARK-5093] Set spark.network.timeout to 120s...

2015-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3903 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-5040][SQL] Support expressing unresolve...

2015-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3862 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-4804] StringContext method to allow Str...

2015-01-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3649#issuecomment-68802004 Now that #3862 has been merged, can we close this issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...

2015-01-05 Thread elmer-garduno
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68785782 Thanks, that worked, I updated the PR to reflect those changes. And here is a list of the actual classes that get included into the jar: jar tf

[GitHub] spark pull request: Support for Mesos DockerInfo

2015-01-05 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/3074#issuecomment-68791982 @ash211 Can you take a look at this patch again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2572] Delete the local dir on executor ...

2015-01-05 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1480#issuecomment-68792088 @watermen Can you update the patch as @andrewor14 mentioned? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3711#discussion_r22495870 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -106,6 +108,7 @@ class

[GitHub] spark pull request: [SPARK-4969][STREAMING][PYTHON] Add binaryReco...

2015-01-05 Thread freeman-lab
Github user freeman-lab commented on the pull request: https://github.com/apache/spark/pull/3803#issuecomment-68797365 Thanks for the review! I'll wait for @JoshRosen 's PR to merge and then update the test here. And will wait for your thoughts on the `getBytes` issue. Otherwise, I

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r22498258 --- Diff: external/kafka/src/main/scala/org/apache/spark/rdd/kafka/KafkaRDD.scala --- @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-5050][Mllib] Add unit test for sqdist

2015-01-05 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3869#discussion_r22498301 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala --- @@ -175,6 +177,33 @@ class VectorsSuite extends FunSuite {

[GitHub] spark pull request: SPARK-4660: Use correct default classloader in...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3840#issuecomment-68785091 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25059/consoleFull) for PR 3840 at commit

[GitHub] spark pull request: SPARK-4660: Use correct default classloader in...

2015-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3840#issuecomment-68785101 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [STREAMING] SPARK-3505: Augmenting SparkStream...

2015-01-05 Thread xiliu82
Github user xiliu82 commented on the pull request: https://github.com/apache/spark/pull/2267#issuecomment-68785881 I will try to do that this week. On Jan 5, 2015, at 11:50 AM, Tathagata Das notificati...@github.com wrote: Ping, for updating this PR.

[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...

2015-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-68787217 [Test build #25060 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25060/consoleFull) for PR 3711 at commit

  1   2   3   4   >