[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu closed the pull request at: https://github.com/apache/spark/pull/18861 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r132881614 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -596,14 +596,17 @@ object CollapseProject extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r132881491 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -596,14 +596,17 @@ object CollapseProject extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131809065 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -596,14 +596,15 @@ object CollapseProject extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131808426 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala --- @@ -746,8 +746,20 @@ abstract class

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131768761 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -596,14 +596,15 @@ object CollapseProject extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131766797 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -2662,6 +2662,30 @@ class Dataset[T] private[sql]( } /**

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131766593 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala --- @@ -746,8 +746,20 @@ abstract class

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131571620 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala --- @@ -753,6 +753,16 @@ case class

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131571547 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -1185,23 +1194,21 @@ class SizeBasedCoalescer(val maxSize: Int) extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131571449 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -1185,23 +1194,21 @@ class SizeBasedCoalescer(val maxSize: Int) extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131571248 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -1185,23 +1194,21 @@ class SizeBasedCoalescer(val maxSize: Int) extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131570851 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -1185,23 +1194,21 @@ class SizeBasedCoalescer(val maxSize: Int) extends

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131570879 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala --- @@ -753,6 +753,16 @@ case class

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131570565 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -571,7 +570,8 @@ case class UnionExec(children:

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18861#discussion_r131570472 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -571,7 +570,8 @@ case class UnionExec(children:

[GitHub] spark pull request #18861: [SPARK-19426][SQL] Custom coalescer for Dataset

GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/18861 [SPARK-19426][SQL] Custom coalescer for Dataset ## What changes were proposed in this pull request? This pr added a new API for `coalesce` in `Dataset`; users can specify the custom coalescer