[GitHub] spark issue #17992: [SPARK-20759] SCALA_VERSION in _config.yml should be con...

2017-05-18 Thread liu-zhaokun
Github user liu-zhaokun commented on the issue: https://github.com/apache/spark/pull/17992 @srowen The test doesn't finish,need I do anything? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request #18015: [SAPRK-20785][WEB-UI][SQL]Spark should provide ju...

2017-05-18 Thread guoxiaolongzte
Github user guoxiaolongzte commented on a diff in the pull request: https://github.com/apache/spark/pull/18015#discussion_r117166546 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala --- @@ -20,7 +20,7 @@ package

[GitHub] spark pull request #18002: [SPARK-20770][SQL] Improve ColumnStats

2017-05-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18002#discussion_r117167072 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala --- @@ -53,219 +53,299 @@ private[columnar] sealed trait

[GitHub] spark pull request #18002: [SPARK-20770][SQL] Improve ColumnStats

2017-05-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18002#discussion_r117167259 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala --- @@ -53,219 +53,299 @@ private[columnar] sealed trait

[GitHub] spark pull request #18002: [SPARK-20770][SQL] Improve ColumnStats

2017-05-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18002#discussion_r117168074 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala --- @@ -53,219 +53,299 @@ private[columnar] sealed trait

[GitHub] spark pull request #18000: [SPARK-20364][SQL] Disable Parquet predicate push...

2017-05-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18000#discussion_r117168737 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -538,6 +538,21 @@ class

[GitHub] spark issue #18011: [SPARK-19089][SQL] Add support for nested sequences

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18011 **[Test build #77040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77040/testReport)** for PR 18011 at commit

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/16989 I think that the current use of `MemoryMode.OFF_HEAP` allocation will cause problems in out-of-the-box deployments using the default configurations. In Spark's current memory manager

[GitHub] spark pull request #18025: [WIP][SparkR] Update doc and examples for sql fun...

2017-05-18 Thread actuaryzhang
GitHub user actuaryzhang opened a pull request: https://github.com/apache/spark/pull/18025 [WIP][SparkR] Update doc and examples for sql functions ## What changes were proposed in this pull request? Create better examples for sql functions. You can merge this pull

[GitHub] spark issue #17997: [SPARK-20763][SQL]The function of `month` and `day` retu...

2017-05-18 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/17997 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

2017-05-18 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/12646#discussion_r117166463 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -510,6 +510,67 @@ public UTF8String trim() { }

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

2017-05-18 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/12646#discussion_r117166374 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -510,6 +510,67 @@ public UTF8String trim() { }

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

2017-05-18 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/12646#discussion_r117166341 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -461,68 +462,246 @@ case class

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

2017-05-18 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/12646#discussion_r117166353 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -1069,6 +1069,8 @@ class AstBuilder(conf: SQLConf)

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

2017-05-18 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/12646#discussion_r117166332 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -461,68 +462,246 @@ case class

[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

2017-05-18 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/12646#discussion_r117167238 --- Diff: common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java --- @@ -730,4 +726,49 @@ public void testToLong() throws

[GitHub] spark pull request #18002: [SPARK-20770][SQL] Improve ColumnStats

2017-05-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18002#discussion_r117168094 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala --- @@ -53,219 +53,299 @@ private[columnar] sealed trait

[GitHub] spark pull request #18000: [SPARK-20364][SQL] Disable Parquet predicate push...

2017-05-18 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18000#discussion_r117168546 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -47,39 +49,47 @@ import

[GitHub] spark issue #18000: [SPARK-20364][SQL] Disable Parquet predicate pushdown fo...

2017-05-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18000 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #17819: [SPARK-20542][ML][SQL] Add a Bucketizer that can bin mul...

2017-05-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/17819 ping @MLnick Do you have more comments on this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16989 **[Test build #77039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77039/testReport)** for PR 16989 at commit

[GitHub] spark issue #18020: [SPARK-20700][SQL] InferFiltersFromConstraints stackover...

2017-05-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18020 Thanks! Merging to master/2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #18020: [SPARK-20700][SQL] InferFiltersFromConstraints st...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18020 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18011: [SPARK-19089][SQL] Add support for nested sequences

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18011 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77040/ Test FAILed. ---

[GitHub] spark issue #18011: [SPARK-19089][SQL] Add support for nested sequences

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18011 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117203261 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java --- @@ -126,4 +150,50 @@ private void

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117169136 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala --- @@ -192,13 +192,7 @@ abstract class HiveComparisonTest

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117172781 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -215,6 +218,217 @@ class StatisticsSuite extends

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117169168 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -232,7 +446,8 @@ class StatisticsSuite extends

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117168812 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -215,6 +218,217 @@ class StatisticsSuite extends

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117156770 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala --- @@ -414,6 +415,50 @@ private[hive] class HiveClientImpl(

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117174283 --- Diff: docs/configuration.md --- @@ -954,12 +971,12 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117174976 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -193,8 +217,19 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117174623 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -193,8 +217,19 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117171649 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -278,4 +278,21 @@ package object config {

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117170538 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java --- @@ -126,4 +150,50 @@ private void

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117172062 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -395,7 +438,6 @@ final class

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117172461 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -129,6 +137,12 @@ final class

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117175176 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -128,41 +133,60 @@ private[spark] class CompressedMapStatus( *

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117170904 --- Diff: core/src/main/scala/org/apache/spark/memory/MemoryManager.scala --- @@ -20,7 +20,7 @@ package org.apache.spark.memory import

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117171397 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -278,4 +278,21 @@ package object config {

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117172780 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -175,33 +197,54 @@ final class

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117170816 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -278,4 +278,21 @@ package object config {

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117170463 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java --- @@ -126,4 +150,50 @@ private void

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117169752 --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java --- @@ -95,6 +97,25 @@ public ManagedBuffer

[GitHub] spark pull request #18002: [SPARK-20770][SQL] Improve ColumnStats

2017-05-18 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18002#discussion_r117177275 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala --- @@ -53,219 +53,299 @@ private[columnar] sealed trait

[GitHub] spark issue #18000: [SPARK-20364][SQL] Disable Parquet predicate pushdown fo...

2017-05-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18000 Seems jenkins doesn't work for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17997: [SPARK-20763][SQL]The function of `month` and `da...

2017-05-18 Thread 10110346
Github user 10110346 commented on a diff in the pull request: https://github.com/apache/spark/pull/17997#discussion_r117180644 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -603,7 +603,13 @@ object DateTimeUtils { */

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16989 +1 on @JoshRosen 's suggestion, we can integrate it with memory manager later. cc @JoshRosen shall we put this patch to branch 2.2? --- If your project is set up for it, you can reply

[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...

2017-05-18 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 I'm working on this now, and am definitely willing to execute the plan we've agreed on, but the more I think about it, the more I think it makes sense to make `ServiceCredentialProvider` private

[GitHub] spark issue #17819: [SPARK-20542][ML][SQL] Add a Bucketizer that can bin mul...

2017-05-18 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/17819 I will try to take a look soon. My main concern is whether we should really have a new class - it starts to make things really messy if we introduce `Multi` versions of everything (e.g. we may want

[GitHub] spark pull request #18022: [SPARK-20790] [MLlib] Correctly handle negative v...

2017-05-18 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18022#discussion_r117191375 --- Diff: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala --- @@ -795,8 +799,8 @@ object ALS extends DefaultParamsReadable[ALS] with

[GitHub] spark pull request #18022: [SPARK-20790] [MLlib] Correctly handle negative v...

2017-05-18 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18022#discussion_r117192700 --- Diff: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala --- @@ -763,11 +763,15 @@ object ALS extends DefaultParamsReadable[ALS] with

[GitHub] spark issue #17094: [SPARK-19762][ML] Hierarchy for consolidating ML aggrega...

2017-05-18 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/17094 In terms of the high level intention of this, agree we definitely need it and it should clean things up substantially. I will start taking a look through ASAP. Thanks! --- If your project is set

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117203833 --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java --- @@ -95,6 +97,25 @@ public ManagedBuffer

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117205121 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -175,33 +197,54 @@ final class

[GitHub] spark issue #18000: [SPARK-20364][SQL] Disable Parquet predicate pushdown fo...

2017-05-18 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18000 LGTM, is parquet going to fix it in the future? or is there any official way to support filter push down for column names with dot? --- If your project is set up for it, you can reply to this

[GitHub] spark issue #18025: [WIP][SparkR] Update doc and examples for sql functions

2017-05-18 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/18025 @felixcheung @HyukjinKwon Per this [suggestion](https://github.com/apache/spark/pull/18003#discussion-diff-116853922L57), I'm creating more meaningful examples for the SQL functions.

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117173550 --- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala --- @@ -29,7 +29,11 @@ import org.apache.spark.shuffle.FetchFailedException

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117175951 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -175,7 +178,7 @@ class StatisticsSuite extends

[GitHub] spark issue #18000: [SPARK-20364][SQL] Disable Parquet predicate pushdown fo...

2017-05-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18000 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #18000: [SPARK-20364][SQL] Disable Parquet predicate pushdown fo...

2017-05-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18000 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117176108 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala --- @@ -175,7 +178,7 @@ class StatisticsSuite extends

[GitHub] spark issue #18012: [SPARK-20779][Examples]The ASF header placed in an incor...

2017-05-18 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18012 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #17997: [SPARK-20763][SQL]The function of `month` and `da...

2017-05-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17997#discussion_r117178267 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -603,7 +603,13 @@ object DateTimeUtils { */

[GitHub] spark pull request #12085: [SPARK-14293] Improve shuffle load balancing and ...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12085 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #12419: [SPARK-14661] [MLlib] trim PCAModel by required e...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12419 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #12162: [SPARK-14289][WIP] Support multiple eviction stra...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12162 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #18022: [SPARK-20790] [MLlib] Correctly handle negative v...

2017-05-18 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18022#discussion_r117191129 --- Diff: mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala --- @@ -78,7 +79,7 @@ class ALSSuite val k = 2 val ne0 =

[GitHub] spark pull request #18022: [SPARK-20790] [MLlib] Correctly handle negative v...

2017-05-18 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18022#discussion_r117190338 --- Diff: mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala --- @@ -348,6 +349,37 @@ class ALSSuite } /** + *

[GitHub] spark pull request #18022: [SPARK-20790] [MLlib] Correctly handle negative v...

2017-05-18 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18022#discussion_r117192420 --- Diff: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala --- @@ -1624,15 +1628,15 @@ object ALS extends DefaultParamsReadable[ALS] with

[GitHub] spark issue #18025: [WIP][SparkR] Update doc and examples for sql functions

2017-05-18 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/18025 This is what the `'column_aggregate_functions.Rd'` doc looks like: ![image](https://cloud.githubusercontent.com/assets/11082368/26190195/fd353224-3b5c-11e7-9a78-2607cc665f49.png)

[GitHub] spark pull request #18002: [SPARK-20770][SQL] Improve ColumnStats

2017-05-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18002#discussion_r117174531 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala --- @@ -53,219 +53,299 @@ private[columnar] sealed trait

[GitHub] spark pull request #17996: [SPARK-20506][DOCS] 2.2 migration guide

2017-05-18 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/17996#discussion_r117174667 --- Diff: docs/ml-guide.md --- @@ -72,35 +72,26 @@ MLlib is under active development. The APIs marked `Experimental`/`DeveloperApi` may change in future

[GitHub] spark issue #18000: [SPARK-20364][SQL] Disable Parquet predicate pushdown fo...

2017-05-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18000 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated S...

2017-05-18 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14971#discussion_r117174966 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala --- @@ -414,6 +415,50 @@ private[hive] class HiveClientImpl(

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/16989 Update: I realize that I overlooked the change to set a default for `spark.memory.offHeap.size`. Thus I'll retract my original objections regarding `MemoryMode.OFF_HEAP` but I'd still like to

[GitHub] spark pull request #17997: [SPARK-20763][SQL]The function of `month` and `da...

2017-05-18 Thread 10110346
Github user 10110346 commented on a diff in the pull request: https://github.com/apache/spark/pull/17997#discussion_r117181219 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -603,7 +603,13 @@ object DateTimeUtils { */

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r117181192 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -193,8 +217,19 @@ private[spark] object HighlyCompressedMapStatus {

[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...

2017-05-18 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17723 @mgummelt We have in house delegation provider for HiveServer2, multi HBase cluster. I think this is useful in Hadoop world. So better to keep it. --- If your project is set up for it, you can

[GitHub] spark issue #16478: [SPARK-7768][SQL] Revise user defined types (UDT)

2017-05-18 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/16478 Is there no SQL committer support for this? Seems like a critical feature for Spark users with no response from any SQL folks. Making UDT public in some way is pretty important no? --- If

[GitHub] spark issue #18000: [SPARK-20364][SQL] Disable Parquet predicate pushdown fo...

2017-05-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18000 Based on the discussion in https://github.com/apache/parquet-mr/pull/361, it does not sound Parquet will support it in the short term. We might need to live with it for a long time. --- If

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/16989 A few more high-level thoughts about this PR: - It seems like the benefits here come from three interrelated changes: - Improving the accuracy of map output size reporting for large

[GitHub] spark pull request #17997: [SPARK-20763][SQL]The function of `month` and `da...

2017-05-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17997#discussion_r117178061 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -603,7 +603,13 @@ object DateTimeUtils { */

[GitHub] spark issue #10405: [SPARK-12339] [WebUI] Added a null check that was remove...

2017-05-18 Thread VishnuGowthemT
Github user VishnuGowthemT commented on the issue: https://github.com/apache/spark/pull/10405 Can this fix be added in 1.6 as well ? https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala --- If your project

[GitHub] spark issue #18024: [SPARK-20792][SS] Support same timeout operations in map...

2017-05-18 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/18024 jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-18 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/16989 Also, I noticed that the PR description doesn't quite align with implementation AFAIK: > Track average size and also the outliers(which are larger than 2*avgSize) in MapStatus;

[GitHub] spark pull request #13762: [SPARK-14926] [ML] OneVsRest labelMetadata uses i...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13762 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #13851: [SPARK-9478] [ml] Add class weights to Random For...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13851 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #16975: [SPARK-19522] Fix executor memory in local-cluste...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16975 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #12491: [SPARK-14712][ML]spark.ml.LogisticRegressionModel...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12491 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #11129: [SPARK-13232][YARN] Fix executor node label

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11129 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #14547: [SPARK-16718][MLlib] gbm-style treeboost

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14547 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #16743: [SPARK-19379][CORE] SparkAppHandle.getState not r...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16743 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #15652: [SPARK-16987] [None] Add spark-default.conf prope...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15652 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #16893: [SPARK-19555][SQL] Improve the performance of Str...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16893 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #17119: [SPARK-19784][SQL][WIP]refresh table after alter ...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17119 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #13881: [SPARK-3723] [MLlib] Adding instrumentation to ra...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13881 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #15914: [SPARK-14974][SQL]delete temporary folder after i...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15914 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #14686: [SPARK-16253][SQL] make spark sql compatible with...

2017-05-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14686 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

  1   2   3   4   5   >