[
https://issues.apache.org/jira/browse/SPARK-60095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126866#comment-18126866
]
Yahya Kisana commented on SPARK-60095:
--------------------------------------
I can take this
> Interpreted MAX/MIN over collated strings return a wrong value
> --------------------------------------------------------------
>
> Key: SPARK-60095
> URL: https://issues.apache.org/jira/browse/SPARK-60095
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 4.2.0, 4.1.1, 5.0.0
> Reporter: Yaoxuan Wu
> Priority: Major
>
> {code:java}
> from pyspark.sql import SparkSession, functions as F
> spark = SparkSession.builder.master("local[1]").getOrCreate() # all rows in
> one partition
> df = spark.createDataFrame([("abcd",), ("xy",), ("ab",)], "s STRING COLLATE
> UTF8_LCASE")
> print(df.agg(F.max("s"), F.min("s")).collect())
> # [Row(max(s)='xy', min(s)='ab')]: correct with
> codegenspark.conf.set("spark.sql.codegen.wholeStage", "false")
> spark.conf.set("spark.sql.codegen.factoryMode", "NO_CODEGEN")
> print(df.agg(F.max("s"), F.min("s")).collect())
> # expected: [Row(max(s)='xy', min(s)='ab')]
> # Spark 4.2.0: [Row(max(s)='ab', min(s)='ab')] {code}
> Same result on 4.1.1, 4.2.0 and master (4.0.0 fails the same way; use a CAST
> to UTF8_BINARY in the output to see it, since collecting a collated column
> returns no rows on 4.0.0). It happens for UTF8_LCASE, UTF8_LCASE_RTRIM and
> UNICODE_CI, not for the default UTF8_BINARY. With more rows the result can
> also be corrupted bytes (we saw 'Ssv\x00'), i.e. the buffer points into
> memory that was overwritten. The rows must be in one partition.
> *Cause:* InternalRow.getWriter copies strings with case StringType => ...
> v.asInstanceOf[UTF8String].copy(). That pattern only matches the default
> StringType object; a collated StringType(id) falls through to case _ =>
> input.update(ordinal, v) and is stored without a copy. Fix: case _:
> StringType =>.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
