[GitHub] spark pull request: [SPARK-4411][UI]Add kill link for jobs in the ...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4823#issuecomment-76586461 [Test build #28140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28140/consoleFull) for PR 4823 at commit [`584240a`](https://gith

[GitHub] spark pull request: [SPARK-4411][UI]Add kill link for jobs in the ...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4823#issuecomment-76586464 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76586319 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76586315 [Test build #28139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28139/consoleFull) for PR 4834 at commit [`65bbbe9`](https://gith

[GitHub] spark pull request: [SPARK-6048] SparkConf should not translate de...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4799#issuecomment-76586005 [Test build #28143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28143/consoleFull) for PR 4799 at commit [`10e77b5`](https://githu

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4835 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5771][UI] Display the actual cores requ...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4841#issuecomment-76584797 [Test build #28138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28138/consoleFull) for PR 4841 at commit [`c44fd6c`](https://gith

[GitHub] spark pull request: [SPARK-5771][UI] Display the actual cores requ...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4841#issuecomment-76584808 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76584666 I'm going to commit this to `master` (1.4.0) in order to fix the bug and failing tests. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-4411][UI]Add kill link for jobs in the ...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4823#issuecomment-76584498 [Test build #28142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28142/consoleFull) for PR 4823 at commit [`25fc0fd`](https://githu

[GitHub] spark pull request: [SPARK-4411][UI]Add kill link for jobs in the ...

2015-02-28 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/4823#issuecomment-76584418 @srowen i have update for your comments. can you take a look again. thanks. --- If your project is set up for it, you can reply to this email and have your reply appe

[GitHub] spark pull request: [SPARK-6088] Correct how tasks that get remote...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4839#issuecomment-76583219 [Test build #28141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28141/consoleFull) for PR 4839 at commit [`f346b49`](https://githu

[GitHub] spark pull request: [SPARK-4411][UI]Add kill link for jobs in the ...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4823#issuecomment-76582533 [Test build #28140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28140/consoleFull) for PR 4823 at commit [`584240a`](https://githu

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76578360 [Test build #28139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28139/consoleFull) for PR 4834 at commit [`65bbbe9`](https://githu

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread MechCoder
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76578092 I changed the randomSplit seed and it works better. It should look good now. --- If your project is set up for it, you can reply to this email and have your reply appea

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4826#issuecomment-76577750 [Test build #28137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28137/consoleFull) for PR 4826 at commit [`d3747d1`](https://gith

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4826#issuecomment-76577757 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-5771][UI] Display the actual cores requ...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4841#issuecomment-76575271 [Test build #28138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28138/consoleFull) for PR 4841 at commit [`c44fd6c`](https://githu

[GitHub] spark pull request: [SPARK-5771][UI] Display the actual cores requ...

2015-02-28 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/4841 [SPARK-5771][UI] Display the actual cores requested when app is finished Display the actual requested cores when app is finished if default core is not set, shows as below: ![image](http

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-28 Thread piaozhexiu
Github user piaozhexiu commented on the pull request: https://github.com/apache/spark/pull/4771#issuecomment-76574787 I've applied the ShutdownHookManager fix to my internal branch and am moving on. I am closing my PR for now. --- If your project is set up for it, you can re

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-28 Thread piaozhexiu
Github user piaozhexiu closed the pull request at: https://github.com/apache/spark/pull/4771 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4826#issuecomment-76572957 [Test build #28137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28137/consoleFull) for PR 4826 at commit [`d3747d1`](https://githu

[GitHub] spark pull request: Fix python typo

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4840#issuecomment-76572162 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: Fix python typo

2015-02-28 Thread robertzk
GitHub user robertzk opened a pull request: https://github.com/apache/spark/pull/4840 Fix python typo You can merge this pull request into a Git repository by running: $ git pull https://github.com/robertzk/spark patch-1 Alternatively you can review and apply these changes as

[GitHub] spark pull request: [SPARK-6055] [PySpark] fix incorrect DataType....

2015-02-28 Thread davies
Github user davies closed the pull request at: https://github.com/apache/spark/pull/4810 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6055] [PySpark] fix incorrect DataType....

2015-02-28 Thread davies
Github user davies closed the pull request at: https://github.com/apache/spark/pull/4809 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6078][CORE] create event log dir if not...

2015-02-28 Thread liyezhang556520
Github user liyezhang556520 closed the pull request at: https://github.com/apache/spark/pull/4829 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featur

[GitHub] spark pull request: [SPARK-6078][CORE] create event log dir if not...

2015-02-28 Thread liyezhang556520
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/4829#issuecomment-76568676 sure, I'll close this PR then, thanks for your review @JoshRosen, @srowen @andrewor14 --- If your project is set up for it, you can reply to this email and hav

[GitHub] spark pull request: SPARK-5984: Fix TimSort bug causes ArrayOutOfB...

2015-02-28 Thread hotou
Github user hotou commented on the pull request: https://github.com/apache/spark/pull/4804#issuecomment-76567920 Thanks for the review guys --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: SPARK-5984: Fix TimSort bug causes ArrayOutOfB...

2015-02-28 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4804#issuecomment-76565164 Thanks. I've merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-5984: Fix TimSort bug causes ArrayOutOfB...

2015-02-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4804 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6088] Correct how tasks that get remote...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4839#issuecomment-76564526 [Test build #28136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28136/consoleFull) for PR 4839 at commit [`84d617c`](https://gith

[GitHub] spark pull request: [SPARK-6088] Correct how tasks that get remote...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4839#issuecomment-76564528 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6088] Correct how tasks that get remote...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4839#issuecomment-76564472 [Test build #28136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28136/consoleFull) for PR 4839 at commit [`84d617c`](https://githu

[GitHub] spark pull request: [SPARK-6088] Correct how tasks that get remote...

2015-02-28 Thread kayousterhout
GitHub user kayousterhout opened a pull request: https://github.com/apache/spark/pull/4839 [SPARK-6088] Correct how tasks that get remote results are shown in UI. It would be great to fix this for 1.3. since the fix is surgical and it helps understandability for users. cc @

[GitHub] spark pull request: [SPARK-6055] [PySpark] fix incorrect DataType....

2015-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4810#issuecomment-76564149 @davies can you close this? (auto close doesn't work for the backport commits). --- If your project is set up for it, you can reply to this email and have your reply ap

[GitHub] spark pull request: [SPARK-6055] [PySpark] fix incorrect DataType....

2015-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4809#issuecomment-76564137 @davies can you close this? (auto close doesn't work for the backport commits). --- If your project is set up for it, you can reply to this email and have your reply ap

[GitHub] spark pull request: [SPARK-6048] SparkConf should not translate de...

2015-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4799#issuecomment-76564115 I agree @andrewor14 can you add documentation about deprecated configs? I would extend what's there now: ``` Properties set directly on the SparkConf

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4838#issuecomment-76563715 Chimed in in a few places. I think overall, a good goal is that when we are doing our normal GC of RDD's and broadcasts, we don't want to be so verbose. This cleaning oc

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4838#discussion_r25563263 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -476,16 +476,16 @@ private[spark] class BlockManagerInfo(

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4838#discussion_r25563255 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1074,7 +1074,7 @@ private[spark] class BlockManager( * Remove all b

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4838#discussion_r25563232 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -188,10 +188,10 @@ private[spark] class ContextCleaner(sc: SparkContext) extends L

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4838#discussion_r25563229 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -371,7 +371,7 @@ private[spark] class MemoryStore(blockManager: BlockManager,

[GitHub] spark pull request: [SPARK-6074] [sql] Package pyspark sql binding...

2015-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4822#issuecomment-76563416 Good catch @vanzin. This LGTM. I did some testing to verify that the assembly includes all relevant python files now: ``` $ jar -tf assembly/target/scala-2.

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4838#issuecomment-76563142 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4838#issuecomment-76563140 [Test build #28135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28135/consoleFull) for PR 4838 at commit [`d9b784d`](https://gith

[GitHub] spark pull request: [SPARK-5253] [ML] LinearRegression with L1/L2 ...

2015-02-28 Thread debasish83
Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/4259#issuecomment-76562346 this is linear regression...what happened to the logistic regression elastic net? We are more interested in that one... --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-5253] [ML] LinearRegression with L1/L2 ...

2015-02-28 Thread debasish83
Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/4259#issuecomment-76562050 @dbtsai we are close to merge this PR which brings OWLQN and PQN under the umbrella of proximal algorithms to support most of the interesting ML related constraints ht

[GitHub] spark pull request: SPARK-2372 [MLLIB] Grouped Optimization/Learni...

2015-02-28 Thread kellrott
Github user kellrott commented on the pull request: https://github.com/apache/spark/pull/1292#issuecomment-76562027 This was originally written as a response to Spark didn't scale to having multiple jobs running at the same time ( http://mail-archives.apache.org/mod_mbox/incuba

[GitHub] spark pull request: [SPARK-3885] Provide mechanism to remove accum...

2015-02-28 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4021#issuecomment-76561166 Thanks for the detailed write up and the fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark pull request: [Streaming][Minor]Fix some error docs in strea...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4837#issuecomment-76561009 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [Streaming][Minor]Fix some error docs in strea...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4837#issuecomment-76561006 [Test build #28134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28134/consoleFull) for PR 4837 at commit [`545291a`](https://gith

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4838#issuecomment-76560597 [Test build #28135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28135/consoleFull) for PR 4838 at commit [`d9b784d`](https://githu

[GitHub] spark pull request: SPARK-3357 [CORE] Internal log messages should...

2015-02-28 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/4838 SPARK-3357 [CORE] Internal log messages should be set at DEBUG level instead of INFO Demote some 'noisy' log messages to debug level. I added a few more, to include everything that gets logged in st

[GitHub] spark pull request: [Streaming][Minor]Fix some error docs in strea...

2015-02-28 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/4837#discussion_r25562626 --- Diff: examples/src/main/python/streaming/kafka_wordcount.py --- @@ -23,7 +23,7 @@ http://kafka.apache.org/documentation.html#quickstart

[GitHub] spark pull request: [Streaming][Minor]Fix some error docs in strea...

2015-02-28 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4837#discussion_r25562444 --- Diff: examples/src/main/python/streaming/kafka_wordcount.py --- @@ -23,7 +23,7 @@ http://kafka.apache.org/documentation.html#quickstart an

[GitHub] spark pull request: [SPARK-6050] [yarn] Add config option to do la...

2015-02-28 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/4818#issuecomment-76556828 I think having the default true would be better so that its backwards compatible. As @sryza mentioned YARN shouldn't really be giving you containers smaller then you

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4826#issuecomment-76556397 [Test build #28132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28132/consoleFull) for PR 4826 at commit [`8360817`](https://gith

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4826#issuecomment-76556398 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [Streaming][Minor]Fix some error docs in strea...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4837#issuecomment-76556189 [Test build #28134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28134/consoleFull) for PR 4837 at commit [`545291a`](https://githu

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76556097 [Test build #28133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28133/consoleFull) for PR 4835 at commit [`4f4b5b2`](https://gith

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76556108 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [Streaming][Minor]Fix some error docs in strea...

2015-02-28 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/4837 [Streaming][Minor]Fix some error docs in streaming examples Small changes, please help to review, thanks a lot. You can merge this pull request into a Git repository by running: $ git pull ht

[GitHub] spark pull request: SPARK-6085 Increase default value for memory o...

2015-02-28 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4836#issuecomment-76554685 I also tend to think that this is a good change, given that a non-trivial proportion of people seem to be running into this and the point was to make it big enough to only

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/4826#discussion_r25562103 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala --- @@ -592,13 +595,79 @@ class MetastoreDataSourcesSuite extend

[GitHub] spark pull request: SPARK-6085 Increase default value for memory o...

2015-02-28 Thread nishkamravi2
Github user nishkamravi2 commented on the pull request: https://github.com/apache/spark/pull/4836#issuecomment-76552283 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark pull request: SPARK-6085 Increase default value for memory o...

2015-02-28 Thread tedyu
Github user tedyu commented on the pull request: https://github.com/apache/spark/pull/4836#issuecomment-76551769 Thanks for the reminder, updated accordingly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: SPARK-6085 Increase default value for memory o...

2015-02-28 Thread nishkamravi2
Github user nishkamravi2 commented on the pull request: https://github.com/apache/spark/pull/4836#issuecomment-76550255 @tedyu We had decided to keep the same default for both yarn and mesos, so it would be best to increase both. Please see mesos/MemoryUtils.scala Also, docs

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76548673 [Test build #28133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28133/consoleFull) for PR 4835 at commit [`4f4b5b2`](https://githu

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76548024 I still think that we should investigate / add that assertion, since correctness of accumulators relies on the assumption that a deserialized task has only one instance

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/4826#discussion_r25561497 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala --- @@ -172,9 +173,14 @@ private[sql] object ParquetRelation { s

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4826#issuecomment-76547402 [Test build #28132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28132/consoleFull) for PR 4826 at commit [`8360817`](https://githu

[GitHub] spark pull request: [SPARK-5950][SQL]Insert array into a metastore...

2015-02-28 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/4826#discussion_r25561465 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/util/modelSaveLoad.scala --- @@ -110,7 +110,7 @@ private[mllib] object Loader { assert(loadedFi

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76546964 That's a legitimate test failure caused by a new assertion that I added here (being extra defensive about avoiding duplicate accumulator registration, since this might

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76546836 [Test build #28131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28131/consoleFull) for PR 4835 at commit [`120c7b0`](https://gith

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76546841 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: SPARK-6085 Increase default value for memory o...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4836#issuecomment-76546540 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: SPARK-6085 Increase default value for memory o...

2015-02-28 Thread tedyu
GitHub user tedyu opened a pull request: https://github.com/apache/spark/pull/4836 SPARK-6085 Increase default value for memory overhead You can merge this pull request into a Git repository by running: $ git pull https://github.com/tedyu/spark master Alternatively you can re

[GitHub] spark pull request: [SPARK-3885] Provide mechanism to remove accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4021#issuecomment-76545044 I have a fix at #4835. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: SPARK-5628 [EC2] Backport: Add version option ...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4833#issuecomment-76544859 [Test build #28129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28129/consoleFull) for PR 4833 at commit [`e723911`](https://gith

[GitHub] spark pull request: SPARK-5628 [EC2] Backport: Add version option ...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4833#issuecomment-76544864 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76544712 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76544707 [Test build #28130 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28130/consoleFull) for PR 4834 at commit [`0c5fe03`](https://gith

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76544337 [Test build #28131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28131/consoleFull) for PR 4835 at commit [`120c7b0`](https://githu

[GitHub] spark pull request: [SPARK-3885] Provide mechanism to remove accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4021#discussion_r25560891 --- Diff: core/src/main/scala/org/apache/spark/Accumulators.scala --- @@ -307,11 +310,22 @@ private[spark] object Accumulators { } }

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76544242 This only affects `master` for the last 6 days or so, by the way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4835#issuecomment-76544221 /cc @andrewor14, it turns out that the "flaky" accumulator test was actually a real bug (fixed by this patch). --- If your project is set up for it, you can reply to t

[GitHub] spark pull request: [SPARK-6075] Fix bug in that caused lost accum...

2015-02-28 Thread JoshRosen
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/4835 [SPARK-6075] Fix bug in that caused lost accumulator updates: do not store WeakReferences in localAccums map This fixes a non-deterministic bug introduced in #4021 that could cause tasks' accumul

[GitHub] spark pull request: [SQL] Checking data types when resolving types

2015-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-76543465 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SQL] Checking data types when resolving types

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4685#issuecomment-76543462 [Test build #28128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28128/consoleFull) for PR 4685 at commit [`a02a169`](https://gith

[GitHub] spark pull request: [SPARK-3885] Provide mechanism to remove accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4021#issuecomment-76543201 > Actually, we won't leak accumulators on executors because we clear the thread-local between tasks. At this point, I should have spotted that it doesn't make s

[GitHub] spark pull request: [SPARK-3885] Provide mechanism to remove accum...

2015-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4021#issuecomment-76542760 I think I've figured it out: consider the lifecycle of an accumulator in a task, say ShuffleMapTask: on the executor, each task deserializes its own copy of the RDD ins

[GitHub] spark pull request: SPARK-5984: Fix TimSort bug causes ArrayOutOfB...

2015-02-28 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/4804#issuecomment-76541409 LGTM as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread MechCoder
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76541233 Hmm. I get an a accuracy of zero for the given example. Not sure where I'm going wrong though :( --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: SPARK-5628 [EC2] Backport: Add version option ...

2015-02-28 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/4833#issuecomment-76541192 LGTM, though I didn't test it out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread MechCoder
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76541036 cc: @mengxr Would you be able to verify this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4834#issuecomment-76541017 [Test build #28130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28130/consoleFull) for PR 4834 at commit [`0c5fe03`](https://githu

[GitHub] spark pull request: [SPARK-6083] [MLLib] [DOC] Make Python API exa...

2015-02-28 Thread MechCoder
GitHub user MechCoder opened a pull request: https://github.com/apache/spark/pull/4834 [SPARK-6083] [MLLib] [DOC] Make Python API example consistent in NaiveBayes You can merge this pull request into a Git repository by running: $ git pull https://github.com/MechCoder/spark sp

[GitHub] spark pull request: SPARK-5628 [EC2] Backport: Add version option ...

2015-02-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4833#issuecomment-76540298 [Test build #28129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28129/consoleFull) for PR 4833 at commit [`e723911`](https://githu

[GitHub] spark pull request: SPARK-5628 [EC2] Backport: Add version option ...

2015-02-28 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/4833 SPARK-5628 [EC2] Backport: Add version option to spark-ec2 Backport of https://github.com/apache/spark/pull/4414 to 1.2: Add version option to spark-ec2 @nchammas @JoshRosen : is this about

  1   2   >