[GitHub] spark issue #14732: [SPARK-16320] [DOC] Document G1 heap region's effect on ...

2016-08-22 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14732 Oh heh, too late. No problem we may further improve the GC docs soon anyway. The existing link wasn't wrong. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #14759: [SPARK-16577][SPARKR] Add CRAN documentation checks to r...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14759 **[Test build #64224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64224/consoleFull)** for PR 14759 at commit

[GitHub] spark issue #14759: [SPARK-16577][SPARKR] Add CRAN documentation checks to r...

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14759 cc @felixcheung @junyangq --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #14759: [SPARK-16577][SPARKR] Add CRAN documentation chec...

2016-08-22 Thread shivaram
GitHub user shivaram opened a pull request: https://github.com/apache/spark/pull/14759 [SPARK-16577][SPARKR] Add CRAN documentation checks to run-tests.sh ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64215/ Test PASSed. ---

[GitHub] spark pull request #12889: [SPARK-15113][PySpark][ML] Add missing num featur...

2016-08-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12889#discussion_r75729345 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -788,6 +788,8 @@ class GeneralizedLinearRegressionModel

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14732: [SPARK-16320] [DOC] Document G1 heap region's effect on ...

2016-08-22 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14732 LGTM. Merging to master and branch 2.0. Thanks @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #64215 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64215/consoleFull)** for PR 14079 at commit

[GitHub] spark pull request #12889: [SPARK-15113][PySpark][ML] Add missing num featur...

2016-08-22 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/12889#discussion_r75728898 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -788,6 +788,8 @@ class

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75728831 --- Diff: docs/tuning.md --- @@ -217,14 +204,22 @@ temporary objects created during task execution. Some steps which may be useful * Check if there are

[GitHub] spark issue #14758: [SPARKR][MINOR] Add Xiangrui and Felix to maintainers

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14758 **[Test build #64223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64223/consoleFull)** for PR 14758 at commit

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75727927 --- Diff: docs/tuning.md --- @@ -217,14 +204,22 @@ temporary objects created during task execution. Some steps which may be useful * Check if there are

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14735 Leaving it out of branch-2.0 sounds good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14758: [SPARKR][MINOR] Add Xiangrui and Felix to maintainers

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14758 cc @mengxr @felixcheung FYI - This is mostly to ensure that we can have more maintainers who can update the CRAN submissions. This shouldn't affect anything else on the development side.

[GitHub] spark pull request #14758: [SPARKR][MINOR] Add Xiangrui and Felix to maintai...

2016-08-22 Thread shivaram
GitHub user shivaram opened a pull request: https://github.com/apache/spark/pull/14758 [SPARKR][MINOR] Add Xiangrui and Felix to maintainers ## What changes were proposed in this pull request? This change adds Xiangrui Meng and Felix Cheung to the maintainers field in the

[GitHub] spark pull request #12889: [SPARK-15113][PySpark][ML] Add missing num featur...

2016-08-22 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12889#discussion_r75727539 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -788,6 +788,8 @@ class GeneralizedLinearRegressionModel

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14753 **[Test build #64213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64213/consoleFull)** for PR 14753 at commit

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-22 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/14735 I don't think they should be required for branch 2.0 - some part of the signature change with ... is likely good to have for consistency but those might also be "breaking" for a *.0.1 release

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14753 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64213/ Test PASSed. ---

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14753 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64211/ Test PASSed. ---

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14753 **[Test build #64211 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64211/consoleFull)** for PR 14753 at commit

[GitHub] spark issue #14757: [SPARK-17190] [SQL] Removal of HiveSharedState

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14757 **[Test build #64222 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64222/consoleFull)** for PR 14757 at commit

[GitHub] spark issue #14750: [SPARK-17183][SQL] put hive serde table schema to table ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14750 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64216/ Test FAILed. ---

[GitHub] spark issue #14750: [SPARK-17183][SQL] put hive serde table schema to table ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14750 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14750: [SPARK-17183][SQL] put hive serde table schema to table ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14750 **[Test build #64216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64216/consoleFull)** for PR 14750 at commit

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75722532 --- Diff: docs/tuning.md --- @@ -122,21 +122,8 @@ large records. `R` is the storage space within `M` where cached blocks immune to being evicted by

[GitHub] spark pull request #14757: [SPARK-17190] [SQL] Removal of HiveSharedState

2016-08-22 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/14757 [SPARK-17190] [SQL] Removal of HiveSharedState ### What changes were proposed in this pull request? Since `HiveClient` is used to interact with the Hive metastore, it should be hidden in

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75722287 --- Diff: docs/tuning.md --- @@ -217,14 +204,22 @@ temporary objects created during task execution. Some steps which may be useful * Check if there are

[GitHub] spark issue #14756: [SPARK-17189][SQL][MINOR] Prefers InternalRow over Unsaf...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14756 **[Test build #64221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64221/consoleFull)** for PR 14756 at commit

[GitHub] spark pull request #14732: [SPARK-16320] [DOC] Document G1 heap region's eff...

2016-08-22 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14732#discussion_r75722253 --- Diff: docs/tuning.md --- @@ -217,14 +204,22 @@ temporary objects created during task execution. Some steps which may be useful * Check if there are

[GitHub] spark pull request #14756: [SPARK-17189][SQL][MINOR] Prefers InternalRow ove...

2016-08-22 Thread clockfly
GitHub user clockfly opened a pull request: https://github.com/apache/spark/pull/14756 [SPARK-17189][SQL][MINOR] Prefers InternalRow over UnsafeRow if UnsafeRow specific interface is not used in AggregationIterator ## What changes were proposed in this pull request? Minor

[GitHub] spark issue #14572: [SPARK-16552] [FOLLOW-UP] [SQL] Store the Inferred Schem...

2016-08-22 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14572 sorry. I missed this PR. Can you update? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14735 @felixcheung I didn't look at the code very closely, but will this change be required in `branch-2.0` as well ? If so the merge might be hard to --- If your project is set up for it, you can

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] doc updates and more CRAN c...

2016-08-22 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75720021 --- Diff: R/pkg/R/DataFrame.R --- @@ -3058,7 +3057,7 @@ setMethod("str", #' @note drop since 2.0.0 setMethod("drop", signature(x =

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] doc updates and more CRAN c...

2016-08-22 Thread junyangq
Github user junyangq commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75719798 --- Diff: R/pkg/NAMESPACE --- @@ -1,5 +1,9 @@ # Imports from base R -importFrom(methods, setGeneric, setMethod, setOldClass) +# Do not include

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] doc updates and more CRAN c...

2016-08-22 Thread junyangq
Github user junyangq commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75719536 --- Diff: R/pkg/R/DataFrame.R --- @@ -3058,7 +3057,7 @@ setMethod("str", #' @note drop since 2.0.0 setMethod("drop", signature(x =

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] doc updates and more CRAN check fi...

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14734 LGTM. I had a couple of minor comments inline. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] doc updates and more CRAN c...

2016-08-22 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75718809 --- Diff: R/pkg/R/generics.R --- @@ -1339,7 +1339,6 @@ setGeneric("spark.naiveBayes", function(data, formula, ...) { standardGeneric("s

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] doc updates and more CRAN c...

2016-08-22 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75718694 --- Diff: R/pkg/R/generics.R --- @@ -1339,7 +1339,6 @@ setGeneric("spark.naiveBayes", function(data, formula, ...) { standardGeneric("s

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] doc updates and more CRAN c...

2016-08-22 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75718243 --- Diff: R/pkg/R/DataFrame.R --- @@ -3058,7 +3057,7 @@ setMethod("str", #' @note drop since 2.0.0 setMethod("drop", signature(x =

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] doc updates and more CRAN c...

2016-08-22 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75717898 --- Diff: R/pkg/NAMESPACE --- @@ -1,5 +1,9 @@ # Imports from base R -importFrom(methods, setGeneric, setMethod, setOldClass) +# Do not include

[GitHub] spark issue #14755: [MINOR][SQL] Fix some typos in comments and test hints

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14755 **[Test build #64219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64219/consoleFull)** for PR 14755 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #64220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64220/consoleFull)** for PR 14079 at commit

[GitHub] spark pull request #14743: [SparkR][Minor] Fix Cache Folder Path in Windows

2016-08-22 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14743 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-22 Thread junyangq
Github user junyangq commented on the issue: https://github.com/apache/spark/pull/14735 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #14755: [MINOR][SQL] Fix some typos in comments and test ...

2016-08-22 Thread clockfly
GitHub user clockfly opened a pull request: https://github.com/apache/spark/pull/14755 [MINOR][SQL] Fix some typos in comments and test hints ## What changes were proposed in this pull request? Fix some typos in comments and test hints ## How was this patch tested?

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 also just realized that I forgot about @kayousterhout 's comment to add in checks on the invariants for the confs -- I've added that now as well. --- If your project is set up for it, you can reply

[GitHub] spark issue #14754: [SPARK-17188][SQL] Moves class QuantileSummaries to proj...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14754 **[Test build #64217 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64217/consoleFull)** for PR 14754 at commit

[GitHub] spark issue #10896: [SPARK-12978][SQL] Skip unnecessary final group-by when ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10896 **[Test build #64218 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64218/consoleFull)** for PR 10896 at commit

[GitHub] spark pull request #14754: [SPARK-17188][SQL] Moves class QuantileSummaries ...

2016-08-22 Thread clockfly
GitHub user clockfly opened a pull request: https://github.com/apache/spark/pull/14754 [SPARK-17188][SQL] Moves class QuantileSummaries to project catalyst for implementing percentile_approx ## What changes were proposed in this pull request? This is a sub-task of

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] doc updates and more CRAN check fi...

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14734 @junyangq Could you take one more look ? I will also do a pass now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14743: [SparkR][Minor] Fix Cache Folder Path in Windows

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14743 BTW LGTM. Merging this PR into master, branch-2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14743: [SparkR][Minor] Fix Cache Folder Path in Windows

2016-08-22 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14743 Thanks @HyukjinKwon -- this is a bit surprising as it was only recently that you fixed the windows tests in https://github.com/apache/spark/commit/1c403733b89258e57daf7b8b0a2011981ad7ed8a

[GitHub] spark pull request #8880: [SPARK-5682][Core] Add encrypted shuffle in spark

2016-08-22 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r75712664 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoConf.scala --- @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #8880: [SPARK-5682][Core] Add encrypted shuffle in spark

2016-08-22 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r75712428 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoConf.scala --- @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark issue #10896: [SPARK-12978][SQL] Skip unnecessary final group-by when ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/10896 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64209/ Test FAILed. ---

[GitHub] spark issue #10896: [SPARK-12978][SQL] Skip unnecessary final group-by when ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/10896 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

2016-08-22 Thread rajeshbalamohan
Github user rajeshbalamohan commented on the issue: https://github.com/apache/spark/pull/14537 For latest ORC, if the data was written out by Hive, it would have the same mapping. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #10896: [SPARK-12978][SQL] Skip unnecessary final group-by when ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10896 **[Test build #64209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64209/consoleFull)** for PR 10896 at commit

[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

2016-08-22 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14537 @rajeshbalamohan So for Orc 2.x files, would schema inference be unnecessary? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #12004: [SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud mo...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12004 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64206/ Test PASSed. ---

[GitHub] spark issue #12004: [SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud mo...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12004 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #12004: [SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud mo...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12004 **[Test build #64206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64206/consoleFull)** for PR 12004 at commit

[GitHub] spark pull request #14298: [SPARK-16283][SQL] Implement `percentile_approx` ...

2016-08-22 Thread clockfly
Github user clockfly commented on a diff in the pull request: https://github.com/apache/spark/pull/14298#discussion_r75709632 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/PercentileApprox.scala --- @@ -0,0 +1,462 @@ +/* + *

[GitHub] spark issue #14750: [SPARK-17183][SQL] put hive serde table schema to table ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14750 **[Test build #64216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64216/consoleFull)** for PR 14750 at commit

[GitHub] spark pull request #10896: [SPARK-12978][SQL] Skip unnecessary final group-b...

2016-08-22 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/10896#discussion_r75707785 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala --- @@ -27,26 +27,87 @@ import

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14239 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14239 **[Test build #64214 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64214/consoleFull)** for PR 14239 at commit

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14239 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64214/ Test FAILed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #64215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64215/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14239 **[Test build #64214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64214/consoleFull)** for PR 14239 at commit

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14753 **[Test build #64213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64213/consoleFull)** for PR 14753 at commit

[GitHub] spark issue #14038: [SPARK-16317][SQL] Add a new interface to filter files i...

2016-08-22 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14038 If my understanding is correct, `PathFilter` is not passed into `FileSystem.listFiles` in `ListingFileCatalog#listLeafFiles` inside. If even so, the performance degrades you pointed out occur? ---

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14239 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64212/ Test FAILed. ---

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14239 **[Test build #64212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64212/consoleFull)** for PR 14239 at commit

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14239 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-22 Thread clockfly
Github user clockfly closed the pull request at: https://github.com/apache/spark/pull/14723 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #10896: [SPARK-12978][SQL] Skip unnecessary final group-b...

2016-08-22 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/10896#discussion_r75701953 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/Aggregate.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #14753: [SPARK-17187][SQL] Supports using arbitrary Java ...

2016-08-22 Thread clockfly
GitHub user clockfly opened a pull request: https://github.com/apache/spark/pull/14753 [SPARK-17187][SQL] Supports using arbitrary Java object as internal aggregation buffer object ## What changes were proposed in this pull request? This PR introduces an abstract class

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14239 **[Test build #64212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64212/consoleFull)** for PR 14239 at commit

[GitHub] spark issue #14723: [SQL][WIP][Test] Supports object-based aggregation funct...

2016-08-22 Thread clockfly
Github user clockfly commented on the issue: https://github.com/apache/spark/pull/14723 @liancheng @cloud-fan @yhuai @hvanhovell @gatorsmile This PR is superceded by #14753, please review the new PR instead. The motivation behind the change is that the aggregation

[GitHub] spark issue #14753: [SPARK-17187][SQL] Supports using arbitrary Java object ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14753 **[Test build #64211 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64211/consoleFull)** for PR 14753 at commit

[GitHub] spark issue #14038: [SPARK-16317][SQL] Add a new interface to filter files i...

2016-08-22 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14038 Oh, i don't want to take on any more work...I just think you should make the predicate passed in something that goes `FileStatus => Boolean` instead of `String => Boolean`, and doing the

[GitHub] spark pull request #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/14079#discussion_r75700225 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -204,6 +213,7 @@ case object TaskResultLost extends TaskFailedReason {

[GitHub] spark issue #14738: [SPARK-17090][FOLLOW-UP][ML]Add expert param support to ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14738 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64208/ Test PASSed. ---

[GitHub] spark issue #14738: [SPARK-17090][FOLLOW-UP][ML]Add expert param support to ...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14738 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14738: [SPARK-17090][FOLLOW-UP][ML]Add expert param support to ...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14738 **[Test build #64208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64208/consoleFull)** for PR 14738 at commit

[GitHub] spark issue #14729: [SPARK-17167] [SQL] Issue Exceptions when Analyze Table ...

2016-08-22 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14729 @viirya Yeah, a normal temporary table would be resolved as a LogicalPlan. Analyze Table does not give us any benefit there. However, you are also allowed to do this: ```sql

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-22 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75700385 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/AggregateWithObjectAggregateBufferSuite.scala --- @@ -0,0 +1,156 @@ +/* + * Licensed to

[GitHub] spark pull request #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-08-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/14079#discussion_r75700278 --- Diff: docs/configuration.md --- @@ -1178,6 +1178,80 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #14749: [SPARK-17182][SQL] Mark Collect as non-determinis...

2016-08-22 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/14749#discussion_r75699347 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala --- @@ -54,6 +54,10 @@ abstract class Collect

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14239 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14239 **[Test build #64210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64210/consoleFull)** for PR 14239 at commit

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14239 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64210/ Test FAILed. ---

[GitHub] spark issue #14749: [SPARK-17182][SQL] Mark Collect as non-deterministic

2016-08-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14749 hmm, I think aggregate function don't need the concept of `deterministic`, as we never check this property for aggregate functions. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2016-08-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14239 **[Test build #64210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64210/consoleFull)** for PR 14239 at commit

[GitHub] spark pull request #10896: [SPARK-12978][SQL] Skip unnecessary final group-b...

2016-08-22 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/10896#discussion_r75695126 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala --- @@ -19,34 +19,90 @@ package

<    1   2   3   4   5   6   7   8   >