[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21895 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21895 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94268/ Test PASSed. ---

[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21895 **[Test build #94268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94268/testReport)** for PR 21895 at commit [`14ae790`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #21988: [SPARK-25003][PYSPARK][BRANCH-2.2] Use SessionExtensions...

2018-08-06 Thread RussellSpitzer
Github user RussellSpitzer commented on the issue: https://github.com/apache/spark/pull/21988 @felixcheung I just didn't know what version to target so I made a a PR for each one. We can just close the ones that shouldn't be merged. --- --

[GitHub] spark issue #21989: [SPARK-25003][PYSPARK][BRANCH-2.3] Use SessionExtensions...

2018-08-06 Thread RussellSpitzer
Github user RussellSpitzer commented on the issue: https://github.com/apache/spark/pull/21989 @kiszk sure, it all depends which branch the merge target should be I wasn't sure which one was being used for changes of this nature. Technically it's a bug fix I believe. --- ---

[GitHub] spark pull request #22004: [WIP][SPARK-25029][TESTS] Scala 2.12 issues: Task...

2018-08-06 Thread adriaanm
Github user adriaanm commented on a diff in the pull request: https://github.com/apache/spark/pull/22004#discussion_r207871809 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2369,39 +2369,12 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark issue #22004: [WIP][SPARK-25029][TESTS] Scala 2.12 issues: TaskNotSeri...

2018-08-06 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/22004 @srowen I was able to reproduce and investigate this further but cleaning in 2.11 does not seem to affect serialization in this case. I moved the serialization check at the beginning of the clean met

[GitHub] spark pull request #22006: [SPARK-25031][SQL] Fix MapType schema print

2018-08-06 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22006#discussion_r207869680 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/types/MapTypeSuite.scala --- @@ -0,0 +1,47 @@ +/* --- End diff -- I think we ca

[GitHub] spark pull request #22004: [WIP][SPARK-25029][TESTS] Scala 2.12 issues: Task...

2018-08-06 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/22004#discussion_r207868702 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2369,39 +2369,12 @@ class DAGSchedulerSuite extends SparkFunSuite wi

[GitHub] spark issue #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row....

2018-08-06 Thread heary-cao
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/21931 @kiszk ,I'm not sure how much the maximum is set, and the size of 1G is the maximum value accepted by numBuckets. Of course, buckets is the memory of 8G. ---

[GitHub] spark issue #21970: [SPARK-24996][SQL] Use DSL in DeclarativeAggregate

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21970 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21970: [SPARK-24996][SQL] Use DSL in DeclarativeAggregate

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21970 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94271/ Test PASSed. ---

[GitHub] spark issue #21970: [SPARK-24996][SQL] Use DSL in DeclarativeAggregate

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21970 **[Test build #94271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94271/testReport)** for PR 21970 at commit [`9103b9c`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21860: [SPARK-24901][SQL]Merge the codegen of RegularHashMap an...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21860 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94270/ Test PASSed. ---

[GitHub] spark issue #21860: [SPARK-24901][SQL]Merge the codegen of RegularHashMap an...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21860 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21860: [SPARK-24901][SQL]Merge the codegen of RegularHashMap an...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21860 **[Test build #94270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94270/testReport)** for PR 21860 at commit [`dfa549e`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #22006: [SPARK-25031][SQL] Fix MapType schema print

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22006 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22006: [SPARK-25031][SQL] Fix MapType schema print

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22006 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22006: [SPARK-25031][SQL] Fix MapType schema print

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22006 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22006: [SPARK-25031][SQL] Fix MapType schema print

2018-08-06 Thread invkrh
GitHub user invkrh opened a pull request: https://github.com/apache/spark/pull/22006 [SPARK-25031][SQL] Fix MapType schema print ## What changes were proposed in this pull request? The PR fix the bug in `buildFormattedString` function in `MapType`, which makes the printed s

[GitHub] spark issue #19635: [SPARK-22413][SQL] Type coercion for IN is not coherent ...

2018-08-06 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19635 kindy ping @gatorsmile @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21184: [WIP][SPARK-24051][SQL] Replace Aliases with the ...

2018-08-06 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21184#discussion_r207859370 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -284,6 +288,80 @@ class Analyzer( } }

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94267/ Test PASSed. ---

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-08-06 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21403 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...

2018-08-06 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/21222#discussion_r207857933 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala --- @@ -513,6 +515,125 @@ class StreamSuite extends StreamTest {

[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #94267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94267/testReport)** for PR 21102 at commit [`33781b6`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94269/ Test FAILed. ---

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21403 **[Test build #94269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94269/testReport)** for PR 21403 at commit [`eb1dfb7`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #21939: [SPARK-23874][SQL][PYTHON] Upgrade Apache Arrow to 0.10....

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21939 It sounds like the vote can pass soon. https://lists.apache.org/thread.html/9900da1540be5aafce27691fd40395bb53f465302db29979c154d99a@%3Cdev.arrow.apache.org%3E --- -

[GitHub] spark pull request #21860: [SPARK-24901][SQL]Merge the codegen of RegularHas...

2018-08-06 Thread heary-cao
Github user heary-cao commented on a diff in the pull request: https://github.com/apache/spark/pull/21860#discussion_r207854432 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala --- @@ -232,6 +232,23 @@ class WholeStageCodegenSuite exten

[GitHub] spark issue #21939: [SPARK-23874][SQL][PYTHON] Upgrade Apache Arrow to 0.10....

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21939 To get this in, we might need to delay the code freeze. Can you reply the dev list email http://apache-spark-developers-list.1001551.n3.nabble.com/code-freeze-and-branch-cut-for-Apache-Spark-2-4

[GitHub] spark issue #21939: [SPARK-23874][SQL][PYTHON] Upgrade Apache Arrow to 0.10....

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21939 After the code freeze, the dependency changes are not allowed. Hopefully, we can make it before that. --- - To unsubscribe,

[GitHub] spark pull request #22000: [SPARK-25025][SQL] Remove the default value of is...

2018-08-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22000 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21909: [SPARK-24959][SQL] Speed up count() for JSON and ...

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21909#discussion_r207850329 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2225,19 +2225,21 @@ class JsonSuite extends Qu

[GitHub] spark issue #21970: [SPARK-24996][SQL] Use DSL in DeclarativeAggregate

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21970 LGTM pending Jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark pull request #21998: [SPARK-24940][SQL] Use IntegerLiteral in ResolveC...

2018-08-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21998 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21986 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207846166 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1930,6 +1930,12 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207845712 --- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala --- @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...

2018-08-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21898 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@sp

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94273/ Test FAILed. ---

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21986 **[Test build #94273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94273/testReport)** for PR 21986 at commit [`37e221c`](https://github.com/apache/spark/commit/3

[GitHub] spark pull request #22001: [SPARK-24819][CORE] Fail fast when no enough slot...

2018-08-06 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/22001#discussion_r207834660 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -496,6 +496,12 @@ class CoarseGrainedSchedu

[GitHub] spark pull request #22001: [SPARK-24819][CORE] Fail fast when no enough slot...

2018-08-06 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/22001#discussion_r207833047 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala --- @@ -453,4 +453,8 @

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-06 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207827294 --- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala --- @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #21087: [SPARK-23997][SQL] Configurable maximum number of bucket...

2018-08-06 Thread ferdonline
Github user ferdonline commented on the issue: https://github.com/apache/spark/pull/21087 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-06 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207823603 --- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala --- @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-06 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207822218 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1930,6 +1930,12 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-06 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207821774 --- Diff: core/src/main/scala/org/apache/spark/BarrierTaskContext.scala --- @@ -39,6 +44,22 @@ class BarrierTaskContext( extends TaskContextImpl(s

[GitHub] spark pull request #21859: [SPARK-24900][SQL]Speed up sort when the dataset ...

2018-08-06 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21859#discussion_r207819972 --- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala --- @@ -166,7 +170,13 @@ class RangePartitioner[K : Ordering : ClassTag, V]( // Ass

[GitHub] spark issue #21087: [SPARK-23997][SQL] Configurable maximum number of bucket...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21087 **[Test build #94274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94274/testReport)** for PR 21087 at commit [`8ddc4eb`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #21087: [SPARK-23997][SQL] Configurable maximum number of bucket...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21087 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94274/ Test FAILed. ---

[GitHub] spark issue #21087: [SPARK-23997][SQL] Configurable maximum number of bucket...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21087 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #22005: [SPARK-16817][CORE][WIP] Use Alluxio to improve stabilit...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22005 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22005: [SPARK-16817][CORE][WIP] Use Alluxio to improve stabilit...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22005 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22005: [SPARK-16817][CORE][WIP] Use Alluxio to improve s...

2018-08-06 Thread Chopinxb
GitHub user Chopinxb opened a pull request: https://github.com/apache/spark/pull/22005 [SPARK-16817][CORE][WIP] Use Alluxio to improve stability of shuffle by replication of shuffle data ## What changes were proposed in this pull request? (In the PR, I propose to use Allux

[GitHub] spark issue #22005: [SPARK-16817][CORE][WIP] Use Alluxio to improve stabilit...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22005 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21087: [SPARK-23997][SQL] Configurable maximum number of bucket...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21087 **[Test build #94274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94274/testReport)** for PR 21087 at commit [`8ddc4eb`](https://github.com/apache/spark/commit/8d

[GitHub] spark pull request #21859: [SPARK-24900][SQL]Speed up sort when the dataset ...

2018-08-06 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21859#discussion_r207817068 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SmallDataSortBenchmark.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed to the

[GitHub] spark pull request #22002: [FOLLOW-UP][SPARK-23772][SQL] Provide an option t...

2018-08-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22002 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21989: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark

2018-08-06 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21989 Is this same as #21990? Would it be possible to close this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21986#discussion_r207816072 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -210,3 +221,66 @@ case class ArrayTransf

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21986 **[Test build #94273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94273/testReport)** for PR 21986 at commit [`37e221c`](https://github.com/apache/spark/commit/37

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1839/

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21087: [SPARK-23997][SQL] Configurable maximum number of...

2018-08-06 Thread ferdonline
Github user ferdonline commented on a diff in the pull request: https://github.com/apache/spark/pull/21087#discussion_r207815199 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -580,6 +580,11 @@ object SQLConf { .booleanConf

[GitHub] spark issue #22002: [FOLLOW-UP][SPARK-23772][SQL] Provide an option to ignor...

2018-08-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22002 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #21898: [SPARK-24817][Core] Implement BarrierTaskContext....

2018-08-06 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21898#discussion_r207814955 --- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala --- @@ -0,0 +1,230 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21986 **[Test build #94272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94272/testReport)** for PR 21986 at commit [`9bbaa3b`](https://github.com/apache/spark/commit/9b

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1838/

[GitHub] spark pull request #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21986#discussion_r207813180 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -210,3 +221,66 @@ case class ArrayTransf

[GitHub] spark pull request #21860: [SPARK-24901][SQL]Merge the codegen of RegularHas...

2018-08-06 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21860#discussion_r207809333 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala --- @@ -232,6 +232,23 @@ class WholeStageCodegenSuite extends

[GitHub] spark issue #21970: [SPARK-24996][SQL] Use DSL in DeclarativeAggregate

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21970 **[Test build #94271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94271/testReport)** for PR 21970 at commit [`9103b9c`](https://github.com/apache/spark/commit/91

[GitHub] spark issue #21970: [SPARK-24996][SQL] Use DSL in DeclarativeAggregate

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21970 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21970: [SPARK-24996][SQL] Use DSL in DeclarativeAggregate

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21970 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1837/

[GitHub] spark pull request #21986: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-06 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21986#discussion_r207808648 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala --- @@ -123,7 +125,10 @@ trait HigherOrderFunct

[GitHub] spark issue #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row....

2018-08-06 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21931 Does this work when we set `30` into the parameter? I am afraid that several arrays with size `0x7fff` are allocated. --- - T

[GitHub] spark issue #21860: [SPARK-24901][SQL]Merge the codegen of RegularHashMap an...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21860 **[Test build #94270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94270/testReport)** for PR 21860 at commit [`dfa549e`](https://github.com/apache/spark/commit/df

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1836/

[GitHub] spark issue #21919: [SPARK-24933][SS] Report numOutputRows in SinkProgress

2018-08-06 Thread vackosar
Github user vackosar commented on the issue: https://github.com/apache/spark/pull/21919 @jose-torres @zsxwing I will exclude SinkProgress constructor from binary compatibility check as this object is constructed internally by Spark. That will remove current MiMa test failure. --- -

[GitHub] spark issue #21860: [SPARK-24901][SQL]Merge the codegen of RegularHashMap an...

2018-08-06 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21860 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark

[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21403 **[Test build #94269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94269/testReport)** for PR 21403 at commit [`eb1dfb7`](https://github.com/apache/spark/commit/eb

[GitHub] spark pull request #21403: [SPARK-24341][SQL] Support only IN subqueries wit...

2018-08-06 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21403#discussion_r207805905 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -505,6 +505,7 @@ object NullPropagation extends Rul

[GitHub] spark pull request #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggrega...

2018-08-06 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21931#discussion_r207803961 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1437,6 +1437,15 @@ object SQLConf { .intConf .crea

[GitHub] spark pull request #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggrega...

2018-08-06 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21931#discussion_r207803415 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1437,6 +1437,15 @@ object SQLConf { .intConf .crea

[GitHub] spark issue #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggregate.row....

2018-08-06 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21931 cc: @cloud-fan @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: re

[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...

2018-08-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21895 **[Test build #94268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94268/testReport)** for PR 21895 at commit [`14ae790`](https://github.com/apache/spark/commit/14

[GitHub] spark pull request #21931: [SPARK-24978][SQL]Add spark.sql.fast.hash.aggrega...

2018-08-06 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21931#discussion_r207802603 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1437,6 +1437,15 @@ object SQLConf { .intConf .cre

[GitHub] spark pull request #21917: [SPARK-24720][STREAMING-KAFKA] add option to alig...

2018-08-06 Thread QuentinAmbard
Github user QuentinAmbard commented on a diff in the pull request: https://github.com/apache/spark/pull/21917#discussion_r207802444 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala --- @@ -223,17 +240,46 @@ private[spar

[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21895 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1835/

[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...

2018-08-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21895 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21917: [SPARK-24720][STREAMING-KAFKA] add option to align range...

2018-08-06 Thread QuentinAmbard
Github user QuentinAmbard commented on the issue: https://github.com/apache/spark/pull/21917 > By failed, you mean returned an empty collection after timing out, even though records should be available? You don't. You also don't know that it isn't just lost because kafka skipped a mes

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21948 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...

2018-08-06 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21895 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21948: [SPARK-24991][SQL] use InternalRow in DataSourceW...

2018-08-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21948#discussion_r207801224 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriterFactory.java --- @@ -33,7 +33,10 @@ public interface DataWriterFac

<    2   3   4   5   6   7   8   >