[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-17 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-55852899 @andrewor14 @JoshRosen I am not sure if the test failure is related to the patch. Can you have a look at the failure? Or just retest it? --- If your project is

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2330#issuecomment-55854376 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20452/consoleFull) for PR 2330 at commit

[GitHub] spark pull request: [SPARK-3491] [WIP] [MLlib] [PySpark] use pickl...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2378#issuecomment-55855022 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/121/consoleFull) for PR 2378 at commit

[GitHub] spark pull request: [SPARK-3491] [WIP] [MLlib] [PySpark] use pickl...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2378#issuecomment-55855160 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20449/consoleFull) for PR 2378 at commit

[GitHub] spark pull request: [SPARK-3477] Clean up code in Yarn Client / Cl...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2350#issuecomment-55855267 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20450/consoleFull) for PR 2350 at commit

[GitHub] spark pull request: [SPARK-3491] [MLlib] [PySpark] use pickle to s...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2378#issuecomment-55855348 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/122/consoleFull) for PR 2378 at commit

[GitHub] spark pull request: [SPARK-3491] [MLlib] [PySpark] use pickle to s...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2378#issuecomment-55855377 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20453/consoleFull) for PR 2378 at commit

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-55856032 This looks like a failure due to a known flaky test: ``` [info] SparkSinkSuite: [info] - Success with ack *** FAILED *** [info] 4000 did not equal

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-55856209 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/123/consoleFull) for PR 1616 at commit

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2330#issuecomment-55856201 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20451/consoleFull) for PR 2330 at commit

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-17 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-55856239 @JoshRosen thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [Docs] Correct spark.files.fetchTimeout defaul...

2014-09-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2406#issuecomment-55856872 LGTM; thanks for updating the title! I'm going to merge this now. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [Docs] Correct spark.files.fetchTimeout defaul...

2014-09-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2406 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2594][SQL] Support CACHE TABLE name A...

2014-09-17 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2397#discussion_r17648914 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala --- @@ -166,3 +166,20 @@ case class DescribeCommand(child: SparkPlan,

[GitHub] spark pull request: [SPARK-2594][SQL] Support CACHE TABLE name A...

2014-09-17 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2397#issuecomment-55857224 LGTM except for the analyzed logical plan issue as mentioned in my last comment. Thanks for working on this! --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SPARK-3319] [SPARK-3338] Resolve Spark submit...

2014-09-17 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/2232#issuecomment-55858105 Could this change behavior in cases where the spark.yarn.dist.files is configured with no scheme? Without this change, it would interpret no scheme to mean that it's on

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2330#issuecomment-55858115 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20452/consoleFull) for PR 2330 at commit

[GitHub] spark pull request: [SPARK-2873] [SQL] using ExternalAppendOnlyMap...

2014-09-17 Thread guowei2
Github user guowei2 commented on the pull request: https://github.com/apache/spark/pull/2029#issuecomment-55858585 I've run a micro benchmark in my local with 5 records,1500 keys. Type | OnHeapAggregate | ExternalAggregate (happens 10 spills) |

[GitHub] spark pull request: [SPARK-3218, SPARK-3219, SPARK-3261, SPARK-342...

2014-09-17 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2419#issuecomment-55858790 @derrickburns I think these notes can go in code comments? (They each generate their own email too.) This is also a big-bang change covering several issues, some

[GitHub] spark pull request: [SPARK-3550][MLLIB] fix a unresolved reference...

2014-09-17 Thread OdinLin
GitHub user OdinLin opened a pull request: https://github.com/apache/spark/pull/2423 [SPARK-3550][MLLIB] fix a unresolved reference variable 'nb' bug variable nb is not reference in the raise log You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [MLLIB] fix a unresolved reference variable 'n...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2423#issuecomment-55860478 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3491] [MLlib] [PySpark] use pickle to s...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2378#issuecomment-55860743 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/122/consoleFull) for PR 2378 at commit

[GitHub] spark pull request: [SPARK-3491] [MLlib] [PySpark] use pickle to s...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2378#issuecomment-55860805 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20453/consoleFull) for PR 2378 at commit

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-55861628 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/123/consoleFull) for PR 1616 at commit

[GitHub] spark pull request: [SPARK-3564] Display App ID on HistoryPage

2014-09-17 Thread sarutak
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/2424 [SPARK-3564] Display App ID on HistoryPage You can merge this pull request into a Git repository by running: $ git pull https://github.com/sarutak/spark display-appid-on-webui Alternatively

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

2014-09-17 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/2330#discussion_r17650877 --- Diff: core/src/main/scala/org/apache/spark/network/netty/BlockClientFactory.scala --- @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2330#discussion_r17650983 --- Diff: core/src/main/scala/org/apache/spark/network/netty/BlockClientFactory.scala --- @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3564] Display App ID on HistoryPage

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2424#issuecomment-55862451 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20454/consoleFull) for PR 2424 at commit

[GitHub] spark pull request: add spark.driver.memory to config docs

2014-09-17 Thread ash211
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55862389 One note about this setting is that I'm not sure it works in all settings -- if you pass it to a driver as a parameter then it's too late to take effect (the JVM has

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread ScrapCodes
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/2425 [SPARK-3543] Write TaskContext in Java and expose it through a static accessor. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651529 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651524 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651575 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2425#issuecomment-55863345 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20455/consoleFull) for PR 2425 at commit

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651616 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651635 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651668 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651760 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651763 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3566] [BUILD] .gitignore and .rat-exclu...

2014-09-17 Thread sarutak
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/2426 [SPARK-3566] [BUILD] .gitignore and .rat-excludes should consider cmd file and Emacs' backup files You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17651774 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-3565]Fix configuration item not consist...

2014-09-17 Thread WangTaoTheTonic
GitHub user WangTaoTheTonic opened a pull request: https://github.com/apache/spark/pull/2427 [SPARK-3565]Fix configuration item not consistent with document https://issues.apache.org/jira/browse/SPARK-3565 spark.ports.maxRetries should be spark.port.maxRetries. Make the

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-55863834 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20456/consoleFull) for PR 2226 at commit

[GitHub] spark pull request: [SPARK-3565]Fix configuration item not consist...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2427#issuecomment-55864303 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20457/consoleFull) for PR 2427 at commit

[GitHub] spark pull request: [SPARK-3566] [BUILD] .gitignore and .rat-exclu...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2426#issuecomment-55864329 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20458/consoleFull) for PR 2426 at commit

[GitHub] spark pull request: SPARK-2745 [STREAMING] Add Java friendly metho...

2014-09-17 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2403#issuecomment-55864499 I'm not sure what to make of the MIMA errors: * the type hierarchy of object org.apache.spark.streaming.Duration has changed in new version. Missing types

[GitHub] spark pull request: [SPARK-3485][SQL] Use GenericUDFUtils.Conversi...

2014-09-17 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2407#issuecomment-55864941 I add a test case to this file, the current master will fail the case because we missed `Short` in the `primitiveTypes` here. With this new solution, we can get rid

[GitHub] spark pull request: [SPARK-3485][SQL] Use GenericUDFUtils.Conversi...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2407#issuecomment-55865221 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20459/consoleFull) for PR 2407 at commit

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

2014-09-17 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/2330#discussion_r17652889 --- Diff: core/src/main/scala/org/apache/spark/network/netty/BlockServer.scala --- @@ -121,18 +91,18 @@ class BlockServer(conf: NettyConfig, dataProvider:

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

2014-09-17 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/2330#discussion_r17653172 --- Diff: core/src/main/scala/org/apache/spark/network/netty/BlockClientFactory.scala --- @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request: https://github.com/apache/spark/pull/2425#discussion_r17653450 --- Diff: core/src/main/java/org/apache/spark/TaskContext.java --- @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: [SPARK-3485][SQL] should check parameter type ...

2014-09-17 Thread adrian-wang
Github user adrian-wang closed the pull request at: https://github.com/apache/spark/pull/2355 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-3485][SQL] should check parameter type ...

2014-09-17 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2355#issuecomment-55867500 maybe 2407 is better, I'll close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2425#issuecomment-55867575 @rxin One side question, Java8ApiSuite(s) don't compile, looks like we have been overlooking them for a while. May be we could just remove them ? --- If your project

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2425#issuecomment-55868820 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20461/consoleFull) for PR 2425 at commit

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-55868821 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20462/consoleFull) for PR 2226 at commit

[GitHub] spark pull request: [SPARK-3566] [BUILD] .gitignore and .rat-exclu...

2014-09-17 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2426#issuecomment-55868873 You can have global gitignore. https://help.github.com/articles/ignoring-files. I am not sure how many editors and such we are going to support. Mind closing this PR.

[GitHub] spark pull request: [SPARK-3564] Display App ID on HistoryPage

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2424#issuecomment-55868956 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20454/consoleFull) for PR 2424 at commit

[GitHub] spark pull request: [SPARK-3565]Fix configuration item not consist...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2427#issuecomment-55869240 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20457/consoleFull) for PR 2427 at commit

[GitHub] spark pull request: [SPARK-3566] [BUILD] .gitignore and .rat-exclu...

2014-09-17 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2426#issuecomment-55869359 @ScrapCodes Thanks for your comment. Vim/vi and Emacs is the most popular editors and current .gitignore considers vim's swp file so Emacs should be considered.

[GitHub] spark pull request: [SPARK-3566] [BUILD] .gitignore and .rat-exclu...

2014-09-17 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2426#issuecomment-55869795 There is nothing like spark-env.cmd in the code base ? + Emacs is my editor of choice too and I add those excludes in gitignore global simply because I can not go and

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2425#issuecomment-55869923 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20455/consoleFull) for PR 2425 at commit

[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-55870259 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20463/consoleFull) for PR 2344 at commit

[GitHub] spark pull request: [SPARK-3566] [BUILD] .gitignore and .rat-exclu...

2014-09-17 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2426#issuecomment-55870586 Current code base doesn't include spark-env.sh but user can create and used by spark-class2.cmd, run-examples2.cmd, compute-classpath.cmd and pyspark2.cmd. And

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-55872750 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20456/consoleFull) for PR 2226 at commit

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2425#issuecomment-55873286 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20464/consoleFull) for PR 2425 at commit

[GitHub] spark pull request: [SPARK-2709] A tool to output all public API o...

2014-09-17 Thread ScrapCodes
Github user ScrapCodes closed the pull request at: https://github.com/apache/spark/pull/1688 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-3485][SQL] Use GenericUDFUtils.Conversi...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2407#issuecomment-55874010 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20459/consoleFull) for PR 2407 at commit

[GitHub] spark pull request: [SPARK-3567] appId field in SparkDeploySchedul...

2014-09-17 Thread sarutak
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/2428 [SPARK-3567] appId field in SparkDeploySchedulerBackend should be volatile You can merge this pull request into a Git repository by running: $ git pull https://github.com/sarutak/spark

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2425#issuecomment-55875586 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20461/consoleFull) for PR 2425 at commit

[GitHub] spark pull request: [SPARK-3567] appId field in SparkDeploySchedul...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2428#issuecomment-55875701 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20465/consoleFull) for PR 2428 at commit

[GitHub] spark pull request: [SPARK-3566] [BUILD] .gitignore and .rat-exclu...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2426#issuecomment-55876167 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20458/consoleFull)** after a configured wait of `120m`. --- If your project

[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-55877253 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20463/consoleFull) for PR 2344 at commit

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-55877476 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20462/consoleFull) for PR 2226 at commit

[GitHub] spark pull request: [SPARK-3543] Write TaskContext in Java and exp...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2425#issuecomment-55879258 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20464/consoleFull) for PR 2425 at commit

[GitHub] spark pull request: [SPARK-3567] appId field in SparkDeploySchedul...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2428#issuecomment-55881519 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20465/consoleFull) for PR 2428 at commit

[GitHub] spark pull request: [SPARK-2594][SQL] Support CACHE TABLE name A...

2014-09-17 Thread ravipesala
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/spark/pull/2397#discussion_r17659871 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala --- @@ -166,3 +166,20 @@ case class DescribeCommand(child: SparkPlan,

[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...

2014-09-17 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2126#issuecomment-55887296 @jongyoul sorry I don't know what you mean by on Aug? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support

2014-09-17 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-55889454 Just rebase code, the failure is in spark-core and not in this patch. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...

2014-09-17 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/2126#issuecomment-55889694 @tgravescs not seriously, I'm deal with this issue unless it's not fixed until Sep. On Wednesday, September 17, 2014, Tom Graves notificati...@github.com

[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...

2014-09-17 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2126#issuecomment-55892735 Ok, sounds good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2778] [yarn] Add yarn integration tests...

2014-09-17 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2257#issuecomment-55895625 @pwendell @andrewor14 Any concerns with the time on this? I know our unit tests already take a long time to run but this doesn't seem to bad. I think going

[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...

2014-09-17 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55895905 @sarutak it is going to be another day or 2 before I can get time to look at this in detail. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: add support for zipping a sequence of RDDs

2014-09-17 Thread mohitjaggi
GitHub user mohitjaggi opened a pull request: https://github.com/apache/spark/pull/2429 add support for zipping a sequence of RDDs a proposed fix for https://issues.apache.org/jira/browse/SPARK-3489 You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: add support for zipping a sequence of RDDs

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2429#issuecomment-55896797 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3218, SPARK-3219, SPARK-3261, SPARK-342...

2014-09-17 Thread derrickburns
Github user derrickburns commented on the pull request: https://github.com/apache/spark/pull/2419#issuecomment-55898846 I agree that some of my comments should go in the code. As for the Big Bang change, I understand your concern. The distance functions touches practically

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1290#issuecomment-55900054 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20466/consoleFull) for PR 1290 at commit

[GitHub] spark pull request: [SPARK-1455] [SPARK-3534] [Build] When possibl...

2014-09-17 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2420#issuecomment-55900507 cc @marmbrus @pwendell Dunno if you guys got notified via JIRA of this pull request or not. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-3535][Mesos] Add 15% task memory overhe...

2014-09-17 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/2401#issuecomment-55902476 Here's a somewhat-related concern: it seems like JVM overhead is unlikely to scale linearly with requested heap size, so maybe a straight-up 15% isn't a great default?

[GitHub] spark pull request: [SPARK-3535][Mesos] Add 15% task memory overhe...

2014-09-17 Thread brndnmtthws
Github user brndnmtthws commented on the pull request: https://github.com/apache/spark/pull/2401#issuecomment-55903580 That implies that as you grow the heap, you're not adding threads (or other things that use off-heap memory). I'm not familiar with Spark's execution model, but I

[GitHub] spark pull request: [SPARK-3535][Mesos] Add 15% task memory overhe...

2014-09-17 Thread brndnmtthws
Github user brndnmtthws commented on the pull request: https://github.com/apache/spark/pull/2401#issuecomment-55903894 Oh, and one more thing you may want to think about is the OS filesystem buffers. Again, as you scale up the heap, you may want to proportionally reserve a slice of

[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...

2014-09-17 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55904701 @tgravescs Thank you so much! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-3069 [DOCS] Build instructions in README...

2014-09-17 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/2014#discussion_r17668529 --- Diff: CONTRIBUTING.md --- @@ -0,0 +1,12 @@ +## Contributing to Spark --- End diff -- Having this is pretty nice! I like the banner you

[GitHub] spark pull request: [Docs] minor grammar fix

2014-09-17 Thread nchammas
GitHub user nchammas opened a pull request: https://github.com/apache/spark/pull/2430 [Docs] minor grammar fix You can merge this pull request into a Git repository by running: $ git pull https://github.com/nchammas/spark patch-2 Alternatively you can review and apply these

[GitHub] spark pull request: [Docs] minor grammar fix

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2430#issuecomment-55906606 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20467/consoleFull) for PR 2430 at commit

[GitHub] spark pull request: add spark.driver.memory to config docs

2014-09-17 Thread nartz
Github user nartz commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55909584 using `./bin/spark-submit --driver-memory 3g myscript.py` on the command line works for me --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: SPARK-3177 (on Master Branch)

2014-09-17 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2204#issuecomment-55910777 looks good. I tested it on hadoop 0.23 and it works also. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: SPARK-3177 (on Master Branch)

2014-09-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2204 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

2014-09-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1290#issuecomment-55911175 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20466/consoleFull) for PR 1290 at commit

[GitHub] spark pull request: [SPARK-2418] Custom checkpointing with an exte...

2014-09-17 Thread darabos
Github user darabos commented on the pull request: https://github.com/apache/spark/pull/1345#issuecomment-55912866 @Forevian, can you please update it to merge cleanly? Then hunt down a reviewer! It would be great to have this in 1.2. It would make our code significantly more

  1   2   3   4   5   >