ulysses-you commented on code in PR #58681:
URL: https://github.com/apache/spark/pull/58681#discussion_r3977724625
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -8185,7 +8204,10 @@ class KeyGroupedPartitioningSuite
withSQLConf(
SQLConf.V2_BUCKETING_SHUFFLE_ENABLED.key -> "true",
"spark.sql.autoBroadcastJoinThreshold" -> "-1",
- SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ // The identity grouping only appears when the marked side is padded
up to the union
+ // of both sides' keys instead of being narrowed to their
intersection.
+ SQLConf.V2_BUCKETING_PARTITION_FILTER_ENABLED.key -> "false") {
Review Comment:
You are right, my comment had the mechanism backwards. 8dcbe648cf6 rewrites
it: with filtering the
grouping is still index-identity, and it is the intersection leaving 2
groups over the marked
side's 3 input partitions that trips the give-up at
`GroupPartitionsExec.scala:101`. The test
predicate now also requires `groupedPartitions.size ==
child.outputPartitioning.numPartitions`,
the second thing `identityGrouping` asks, so the predicate no longer
over-claims. The pin stays
for now, for the reason in my reply to Finding 2.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2547,7 +2547,7 @@ object SQLConf {
s"enabled.")
.version("4.0.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
Review Comment:
Agreed on the whole read, including that correctness rests on
`areKeysCompatible`'s subset rule
rather than on the give-up, and that the cost is AQE rules silently skipping
the stage.
Decision: this PR keeps all three flips and adds no guard. SPARK-59396
targets 4.4.0, so it can
land after SPARK-59272 and your #58659 line.
##########
docs/sql-performance-tuning.md:
##########
@@ -688,6 +688,46 @@ The following SQL properties enable Storage Partition Join
in different join que
</td>
<td>4.0.0</td>
</tr>
+ <tr>
+
<td><code>spark.sql.sources.v2.bucketing.partition.filter.enabled</code></td>
+ <td>true</td>
+ <td>
+ When enabled, key groups that cannot produce output for the join type
are not scanned at all, instead of being filled with empty partitions on the
side that does not hold them. For example, an inner join only scans the key
groups present on both sides. This config requires both
<code>spark.sql.sources.v2.bucketing.enabled</code> and
<code>spark.sql.sources.v2.bucketing.pushPartValues.enabled</code> to be true.
+ </td>
+ <td>4.0.0</td>
+ </tr>
+ <tr>
+ <td><code>spark.sql.sources.v2.bucketing.sorting.enabled</code></td>
+ <td>false</td>
+ <td>
+ When enabled, Spark satisfies a sort on the partition key expressions
from the partitioning reported by a V2 data source, so no shuffle is added for
that sort. The parallelism of the sorted output is then whatever the data
source's partition layout provides, and there is no shuffle stage left for
adaptive partition coalescing or skew splitting to balance. This config
requires <code>spark.sql.sources.v2.bucketing.enabled</code> to be true.
+ </td>
+ <td>4.0.0</td>
+ </tr>
+ <tr>
+
<td><code>spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled</code></td>
+ <td>true</td>
+ <td>
+ When enabled, Spark derives the output ordering of a V2 scan from its
partition key expressions, if the source reports a keyed partitioning but no
explicit ordering. All rows of such a partition share one key value, so the
partition is trivially sorted by those expressions, and a sort Spark would
otherwise add becomes unnecessary. This config requires
<code>spark.sql.sources.v2.bucketing.enabled</code> to be true.
+ </td>
+ <td>4.2.0</td>
+ </tr>
+ <tr>
+
<td><code>spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled</code></td>
+ <td>true</td>
+ <td>
+ When enabled, <code>GroupPartitionsExec</code> reports sort orders
over partition key expressions after coalescing several input partitions into
one. The merged partitions share the same partition key value, so these orders
still hold, while orders over other columns are lost by the concatenation. This
config requires <code>spark.sql.sources.v2.bucketing.enabled</code> to be true.
+ </td>
+ <td>4.2.0</td>
+ </tr>
+ <tr>
+
<td><code>spark.sql.sources.v2.bucketing.preserveOrderingOnCoalesce.enabled</code></td>
+ <td>false</td>
+ <td>
+ When enabled, <code>GroupPartitionsExec</code> merges partitions that
share a key by a sorted merge rather than by concatenation, so it can report
the child's full ordering instead of only the orderings over partition key
expressions that
<code>spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled</code>
preserves. This removes a downstream sort when data is both partitioned and
sorted, but a sorted merge costs more than concatenation, especially when
merging many partitions. This config requires
<code>spark.sql.sources.v2.bucketing.enabled</code> to be true.
Review Comment:
Adopted your wording verbatim in 8dcbe648cf6.
##########
docs/sql-performance-tuning.md:
##########
@@ -688,6 +688,46 @@ The following SQL properties enable Storage Partition Join
in different join que
</td>
<td>4.0.0</td>
</tr>
+ <tr>
+
<td><code>spark.sql.sources.v2.bucketing.partition.filter.enabled</code></td>
+ <td>true</td>
+ <td>
+ When enabled, key groups that cannot produce output for the join type
are not scanned at all, instead of being filled with empty partitions on the
side that does not hold them. For example, an inner join only scans the key
groups present on both sides. This config requires both
<code>spark.sql.sources.v2.bucketing.enabled</code> and
<code>spark.sql.sources.v2.bucketing.pushPartValues.enabled</code> to be true.
Review Comment:
Adopted, and fixed at the source too: the same "requires both" wording in the
`V2_BUCKETING_PARTITION_FILTER_ENABLED` doc string now reads `enabled`
together with either
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` or
`spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
