[
https://issues.apache.org/jira/browse/SPARK-58928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod KC updated SPARK-58928:
-----------------------------
Description:
For a query like _RANK() OVER (PARTITION BY s ...)_ with {_}WHERE rank <= N{_},
the optimizer adds a WindowGroupLimitExec to drop rows early. It finds
partition boundaries by comparing keys byte-for-byte (nextGroup ==
currentGroup).
Byte comparison is wrong for collated strings: under UTF8_LCASE, 'foo' and
'FOO' are equal but have different bytes, so one partition gets split into
several.
The final result is still correct (the later WindowExec recomputes the
ranking), but the limit no longer prunes collated partitions, so it forwards
many more rows than needed and the optimization stops helping.
Repro:
{code:java}
CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
SELECT s, i, r FROM (
SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
{code}
Need to compare keys the same way WindowExec already does, use byte comparison
only when all keys are binary-stable, and fall back to collation-aware
comparison when any key is not.
was:
For a query like _RANK() OVER (PARTITION BY s ...)_ with {_}WHERE rank <= N{_},
the optimizer adds a WindowGroupLimitExec to drop rows early. It finds
partition boundaries by comparing keys byte-for-byte (nextGroup ==
currentGroup).
Byte comparison is wrong for collated strings: under UTF8_LCASE, 'foo' and
'FOO' are equal but have different bytes, so one partition gets split into
several.
The final result is still correct (the later WindowExec recomputes the
ranking), but the limit no longer prunes collated partitions, so it forwards
many more rows than needed and the optimization stops helping.
Repro:
{code:java}
CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
SELECT s, i, r FROM (
SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
{code}
Need to compare keys the same way WindowExec already does, use byte comparison
only when all keys are binary-stable, and fall back to collation-aware
comparison when any key is not.
> WindowGroupLimitExec uses binary equality for collated PARTITION BY keys
> ------------------------------------------------------------------------
>
> Key: SPARK-58928
> URL: https://issues.apache.org/jira/browse/SPARK-58928
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Vinod KC
> Priority: Major
>
> For a query like _RANK() OVER (PARTITION BY s ...)_ with {_}WHERE rank <=
> N{_}, the optimizer adds a WindowGroupLimitExec to drop rows early. It finds
> partition boundaries by comparing keys byte-for-byte (nextGroup ==
> currentGroup).
> Byte comparison is wrong for collated strings: under UTF8_LCASE, 'foo' and
> 'FOO' are equal but have different bytes, so one partition gets split into
> several.
> The final result is still correct (the later WindowExec recomputes the
> ranking), but the limit no longer prunes collated partitions, so it forwards
> many more rows than needed and the optimization stops helping.
> Repro:
> {code:java}
> CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
> INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
> SELECT s, i, r FROM (
> SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
> ) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
>
> {code}
> Need to compare keys the same way WindowExec already does, use byte
> comparison only when all keys are binary-stable, and fall back to
> collation-aware comparison when any key is not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
