[GitHub] spark pull request #16193: [SPARK-18766] [SQL] Push Down Filter Through Batc...

2016-12-08 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/16193#discussion_r91609923 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala --- @@ -165,4 +167,31 @@ object ExtractPythonUDFs extends Rul

[GitHub] spark issue #16200: [SPARK-18773][core] Make commons-crypto config translati...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16200 **[Test build #69880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69880/consoleFull)** for PR 16200 at commit [`0dc4c94`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #16154: [SPARK-17822] [R] Make JVMObjectTracker a member variabl...

2016-12-08 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/16154 Confirmed this patch fixed the issue in our environment:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #16218: [SPARKR][SPARK-18590] Change the R source build t...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16218 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16219: [SPARK-18790][SS] Keep a general offset history of strea...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16219 **[Test build #69879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69879/consoleFull)** for PR 16219 at commit [`fc1557e`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #16218: Change the R source build to Hadoop 2.6

2016-12-08 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16218 No worries. Since this isn't tested by jenkins, I'm going to merge this and test this manually --- If your project is set up for it, you can reply to this email and have your reply appear on GitHu

[GitHub] spark pull request #16219: [SPARK-18790][SS] Keep a general offset history o...

2016-12-08 Thread tcondie
GitHub user tcondie opened a pull request: https://github.com/apache/spark/pull/16219 [SPARK-18790][SS] Keep a general offset history of stream batches ## What changes were proposed in this pull request? Instead of only keeping the minimum number of offsets around, we should

[GitHub] spark pull request #16191: spark decision tree

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16191 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #14561: [SPARK-16972][CORE] Move DriverEndpoint out of Co...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14561 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #13026: [SPARK-15218][DEPLOY] replace any occurence of ':...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13026 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #16198: adding yarn module in pom

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16198 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #7739: [SPARK-9384] [core] Easier setting of executor and...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7739 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request #14223: [SPARK-10614][CORE] Change SystemClock to derive ...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14223 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #16217: Close stale PRs.

2016-12-08 Thread vanzin
Github user vanzin closed the pull request at: https://github.com/apache/spark/pull/16217 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16217: Close stale PRs.

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16217 Ah, cool. I was planning to do that later, but since you already did, closing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16217: Close stale PRs.

2016-12-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16217 Similar to the last pr i coudln't merge this with merge tool but I pushed directly to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16154: [SPARK-17822] [R] Make JVMObjectTracker a member variabl...

2016-12-08 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/16154 @yhuai @jkbradley I'm testing the patch in our environment to see whether it actually solves the problem. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16216: [SPARK-18774][CORE][SQL] Ignore non-existing files when ...

2016-12-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16216 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #16187: [SPARK-18760][SQL] Consistent format specificatio...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16187 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #14680: [SPARK-17101][SQL] Provide consistent format iden...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14680 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #16193: [SPARK-18766] [SQL] Push Down Filter Through Batc...

2016-12-08 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16193#discussion_r91604263 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala --- @@ -166,3 +174,40 @@ object ExtractPythonUDFs extends

[GitHub] spark issue #16193: [SPARK-18766] [SQL] Push Down Filter Through BatchEvalPy...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16193 **[Test build #69878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69878/consoleFull)** for PR 16193 at commit [`3d9ba67`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #16187: [SPARK-18760][SQL] Consistent format specification for F...

2016-12-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16187 Fixed- let me merge this in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark issue #16218: Change the R source build to Hadoop 2.6

2016-12-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16218 LGTM. Sorry about that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark issue #16200: [SPARK-18773][core] Make commons-crypto config translati...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16200 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69867/ Test FAILed. ---

[GitHub] spark issue #16200: [SPARK-18773][core] Make commons-crypto config translati...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16200 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers example cod...

2016-12-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16148 Agreed. We will definitely keep that in mind but there is certainly a lot going in working the API already. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #16200: [SPARK-18773][core] Make commons-crypto config translati...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16200 **[Test build #69867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69867/consoleFull)** for PR 16200 at commit [`cf32d3c`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15219: [SPARK-14098][SQL] Generate Java code to build CachedCol...

2016-12-08 Thread velvia
Github user velvia commented on the issue: https://github.com/apache/spark/pull/15219 Hi, in my experience, interfaces which are implemented by final methods are inlined just fine in the JVM JIT…. but you might know more about this. The point is more simplification and that the

[GitHub] spark issue #16217: Close stale PRs.

2016-12-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16217 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13579 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69868/ Test FAILed. ---

[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13579 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13579 **[Test build #69868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69868/consoleFull)** for PR 13579 at commit [`f375682`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15219: [SPARK-14098][SQL] Generate Java code to build CachedCol...

2016-12-08 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/15219 @velvia Thank you for your comment. It is interesting from the view of simplification. On the other hand, I think that current implementation thinks about performance to encourage method inlining

[GitHub] spark issue #15275: [SPARK-17702][SQL] Code generation including too many mu...

2016-12-08 Thread rezasafi
Github user rezasafi commented on the issue: https://github.com/apache/spark/pull/15275 Any update on the decision of whether this fix will be backported to 1.6? thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #16218: Change the R source build to Hadoop 2.6

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16218 **[Test build #69877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69877/consoleFull)** for PR 16218 at commit [`abae93c`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #16218: Change the R source build to Hadoop 2.6

2016-12-08 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16218 cc @felixcheung @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #16218: Change the R source build to Hadoop 2.6

2016-12-08 Thread shivaram
GitHub user shivaram opened a pull request: https://github.com/apache/spark/pull/16218 Change the R source build to Hadoop 2.6 This PR changes the SparkR source release tarball to be built using the Hadoop 2.6 profile. Previously it was using the without hadoop profile which leads

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

2016-12-08 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 @felixcheung I triggered a nightly build on branch-2.1 and this doesn't work correctly in the no-hadoop build. While building the vignettes we run into an error -- I'm going to change this to use a

[GitHub] spark issue #16195: [Spark-18765] [CORE] Make values for spark.yarn.{am|driv...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16195 **[Test build #69875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69875/consoleFull)** for PR 16195 at commit [`2f2a6e3`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #16043: [SPARK-18601][SQL] Simplify Create/Get complex expressio...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16043 **[Test build #69876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69876/consoleFull)** for PR 16043 at commit [`7ddc5db`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #16043: [SPARK-18601][SQL] Simplify Create/Get complex expressio...

2016-12-08 Thread eyalfa
Github user eyalfa commented on the issue: https://github.com/apache/spark/pull/16043 @HyukjinKwon , I think I've handled all of your comments but two: 1. the one with pattern matching on `elems` (please see comment) 2. use of the Matchers syntax in tests, please respond to comm

[GitHub] spark issue #16214: [SPARK-18325][SPARKR] Add example for using native R pac...

2016-12-08 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/16214 @yanboliang What happens if there are multiple executors on the same machine? Are there concurrency issues? --- If your project is set up for it, you can reply to this email and have your reply appe

[GitHub] spark issue #16205: [SPARK-18776][SS] Make Offset for FileStreamSource corre...

2016-12-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16205 Oh, please also support to read the old format in case RC2 is passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #16043: [SPARK-18601][SQL] Simplify Create/Get complex ex...

2016-12-08 Thread eyalfa
Github user eyalfa commented on a diff in the pull request: https://github.com/apache/spark/pull/16043#discussion_r91596315 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala --- @@ -0,0 +1,482 @@ +/* + * Licensed to the A

[GitHub] spark issue #16205: [SPARK-18776][SS] Make Offset for FileStreamSource corre...

2016-12-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16205 LGTM. Merging to master and 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark pull request #16043: [SPARK-18601][SQL] Simplify Create/Get complex ex...

2016-12-08 Thread eyalfa
Github user eyalfa commented on a diff in the pull request: https://github.com/apache/spark/pull/16043#discussion_r91595458 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala --- @@ -0,0 +1,131 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #16217: Close stale PRs.

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16217 **[Test build #69874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69874/consoleFull)** for PR 16217 at commit [`dee1efe`](https://github.com/apache/spark/commit/d

[GitHub] spark pull request #16178: [SPARK-18751][Core]Fix deadlock when SparkContext...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16178 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16217: Close stale PRs.

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16217 Let's see if others have anything to add. @srowen @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #16217: Close stale PRs.

2016-12-08 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/16217 Close stale PRs. Closes #16191 Closes #16198 Closes #14561 Closes #14223 Closes #7739 Closes #13026 You can merge this pull request into a Git repository by running: $ git p

[GitHub] spark issue #16178: [SPARK-18751][Core]Fix deadlock when SparkContext.stop i...

2016-12-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16178 Thanks! Merging to master and 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13026: [SPARK-15218][DEPLOY] replace any occurence of ':' in SP...

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/13026 I'm gonna propose we close this because (1) this is an old Mesos version being terribly broken and (2) spark-class is not expected to be able to write to `$SPARK_HOME`, much less its parent, so the f

[GitHub] spark issue #16148: [SPARK-18325][SparkR][ML] SparkR ML wrappers example cod...

2016-12-08 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16148 @felixcheung This is fine to merge since it is for docs/examples only. But in general, we should be better about adding these right away after adding the R APIs, rather than doing them all at the

[GitHub] spark pull request #16195: [Spark-18765] [CORE] Make values for spark.yarn.{...

2016-12-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16195#discussion_r91592278 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -61,11 +61,15 @@ private[spark] class ClientArguments(args: Array[Str

[GitHub] spark issue #15787: [SPARK-18286][ML] Add Scala/Java examples for MinHash an...

2016-12-08 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/15787 Ah, thank you @bravo-zhang ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16176: [SPARK-18746][SQL] Add newBigDecimalEncoder

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16176 **[Test build #69873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69873/consoleFull)** for PR 16176 at commit [`8d23a00`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #15219: [SPARK-14098][SQL] Generate Java code to build CachedCol...

2016-12-08 Thread velvia
Github user velvia commented on the issue: https://github.com/apache/spark/pull/15219 @kiszk I had a quick look at your `ColumnarVector` and had some thoughts. Right now, it seems `ColumnVector` as an abstract class mixes lots of functionalities into one. - Methods fo

[GitHub] spark pull request #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter met...

2016-12-08 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16017#discussion_r91591028 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala --- @@ -52,33 +52,49 @@ class DecisionTreeClassifier @Since

[GitHub] spark pull request #16195: [Spark-18765] [CORE] Make values for spark.yarn.{...

2016-12-08 Thread daisukebe
Github user daisukebe commented on a diff in the pull request: https://github.com/apache/spark/pull/16195#discussion_r91589664 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -61,11 +61,15 @@ private[spark] class ClientArguments(args: Array[

[GitHub] spark issue #16209: [WIP][SPARK-10849][SQL] Adds option to the JDBC data sou...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16209 **[Test build #69871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69871/consoleFull)** for PR 16209 at commit [`faa8172`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #16176: [SPARK-18746][SQL] Add newBigDecimalEncoder

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16176 **[Test build #69872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69872/consoleFull)** for PR 16176 at commit [`3e6e8fa`](https://github.com/apache/spark/commit/3

[GitHub] spark pull request #16014: [SPARK-18590][SPARKR] build R source package when...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16014 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16209: [WIP][SPARK-10849][SQL] Adds option to the JDBC data sou...

2016-12-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16209 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

2016-12-08 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 LGTM. I took another look at `release-build.sh` and I think it looks fine. Merging this into master, branch-2.1. I'll also see if I can test this out somehow on jenkins --- If your project is se

[GitHub] spark pull request #13579: [SPARK-15844] [core] HistoryServer doesn't come u...

2016-12-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/13579#discussion_r91587654 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala --- @@ -349,6 +349,17 @@ class HistoryServerSuite extends SparkFunSuite

[GitHub] spark issue #16154: [SPARK-17822] [R] Make JVMObjectTracker a member variabl...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16154 **[Test build #69870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69870/consoleFull)** for PR 16154 at commit [`efe332d`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #16164: [SPARK-18732][WEB-UI] The Y axis ranges of "schedulingDe...

2016-12-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16164 It's on purpose. > It still may misleads users in some cases, like 'scheduling delay' in 'ms' and 'processing time' in 's' or 'min' I don't get it. I saw the time units in your scre

[GitHub] spark issue #16216: [SPARK-18774][CORE][SQL] Ignore non-existing files when ...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16216 **[Test build #69869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69869/consoleFull)** for PR 16216 at commit [`841ac5d`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #16154: [SPARK-17822] [R] Make JVMObjectTracker a member variabl...

2016-12-08 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/16154 @falaki I'm happy to add the first Scala unit test for `RBackend`:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark pull request #16154: [SPARK-17822] [R] Make JVMObjectTracker a member ...

2016-12-08 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/16154#discussion_r91586125 --- Diff: core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala --- @@ -143,12 +142,8 @@ private[r] class RBackendHandler(server: RBackend)

[GitHub] spark issue #16203: [SPARK-18774][Core][SQL]Ignore non-existing files when i...

2016-12-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16203 Submitted #16216 to backport it to 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature ena

[GitHub] spark pull request #16216: [SPARK-18774][CORE][SQL] Ignore non-existing file...

2016-12-08 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/16216 [SPARK-18774][CORE][SQL] Ignore non-existing files when ignoreCorruptFiles is enabled (branch 2.1) ## What changes were proposed in this pull request? Backport #16203 to branch 2.1.

[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...

2016-12-08 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/13579 done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #16121: [SPARK-16589][PYTHON] Chained cartesian produces ...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16121 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16121: [SPARK-16589][PYTHON] Chained cartesian produces incorre...

2016-12-08 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/16121 LGTM, merging into master and 2.1 branch, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark issue #16193: [SPARK-18766] [SQL] Push Down Filter Through BatchEvalPy...

2016-12-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16193 ExistingRDD might not be always the child of `Filter`. For example, ```Python >>> sel = df.select('key', 'value', rand()).filter((my_filter(col("key"))) & (df.value < "2")) ``` ``

[GitHub] spark pull request #16014: [SPARK-18590][SPARKR] build R source package when...

2016-12-08 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/16014#discussion_r91583848 --- Diff: dev/create-release/release-build.sh --- @@ -172,11 +172,30 @@ if [[ "$1" == "package" ]]; then MVN_HOME=`$MVN -version 2>&1 | grep 'Maven

[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/13579 While tests are running, can you update the description? It mentions env vars, which is misleading since they don't have anything to do with this. --- If your project is set up for it, you can reply

[GitHub] spark pull request #16014: [SPARK-18590][SPARKR] build R source package when...

2016-12-08 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/16014#discussion_r91579011 --- Diff: dev/create-release/release-build.sh --- @@ -221,14 +235,13 @@ if [[ "$1" == "package" ]]; then # We increment the Zinc port each time

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16213 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16213 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69863/ Test FAILed. ---

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16213 **[Test build #69863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69863/consoleFull)** for PR 16213 at commit [`e8a4e1d`](https://github.com/apache/spark/commit/

[GitHub] spark issue #16150: [SPARK-18349][SparkR]:Update R API documentation on ml m...

2016-12-08 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16150 Yeah. This is mostly docs fixes - I think this is fine to go into branch-2.1. We can see which RC gets promoted etc. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #16165: [SPARK-8617] [WEBUI] HistoryServer: Include in-pr...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16165 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16165: [SPARK-8617] [WEBUI] HistoryServer: Include in-progress ...

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16165 Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13579 **[Test build #69868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69868/consoleFull)** for PR 13579 at commit [`f375682`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #13579: [SPARK-15844] [core] HistoryServer doesn't come up if sp...

2016-12-08 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/13579 stylecheck; unexpected, as I thought I'd run them in the `mvn install` of the module. ``` [error] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/test/scala/org/apache/spark/d

[GitHub] spark issue #16165: [SPARK-8617] [WEBUI] HistoryServer: Include in-progress ...

2016-12-08 Thread seyfe
Github user seyfe commented on the issue: https://github.com/apache/spark/pull/16165 Thank @vanzin. Updated the description based on final state. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request #16068: [SPARK-18637][SQL]Stateful UDF should be consider...

2016-12-08 Thread zhzhan
Github user zhzhan commented on a diff in the pull request: https://github.com/apache/spark/pull/16068#discussion_r91569919 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala --- @@ -487,6 +489,26 @@ class HiveUDFSuite extends QueryTest with

[GitHub] spark issue #16200: [SPARK-18773][core] Make commons-crypto config translati...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16200 **[Test build #69867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69867/consoleFull)** for PR 16200 at commit [`cf32d3c`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #16165: [SPARK-8617] [WEBUI] HistoryServer: Include in-progress ...

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16165 Code LGTM but can you update the description to reflect the recent changes to the PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark issue #16000: [SPARK-18537][Web UI]Add a REST api to spark streaming

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16000 If you're not willing to make the changes, we should close this one, if you don't mind, and reopen @uncleGen's PR instead. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request #16202: [SPARK-18662][hotfix] Add new resource-managers d...

2016-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16202 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16200: [SPARK-18773][core] Make commons-crypto config translati...

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16200 hmm. retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16202: [SPARK-18662][hotfix] Add new resource-managers director...

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16202 Merging to master to make the builds happy. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16195: [Spark-18765] [CORE] Make values for spark.yarn.{am|driv...

2016-12-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16195 **[Test build #69866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69866/consoleFull)** for PR 16195 at commit [`10e0c75`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #16202: [SPARK-18662][hotfix] Add new resource-managers director...

2016-12-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16202 Yeah, mesos was missing. But I guess that only affects a tiny number of people who'd like to use SPARK_PREPEND_CLASSES while changing that mesos code, so probably nobody noticed. --- If your projec

[GitHub] spark issue #16121: [SPARK-16589][PYTHON] Chained cartesian produces incorre...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16121 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16121: [SPARK-16589][PYTHON] Chained cartesian produces incorre...

2016-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16121 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69865/ Test PASSed. ---

<    1   2   3   4   5   6   >