Bruce Robbins created SPARK-46289:
-------------------------------------
Summary: Exception when ordering by UDT in interpreted mode
Key: SPARK-46289
URL: https://issues.apache.org/jira/browse/SPARK-46289
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.0, 3.4.2
Reporter: Bruce Robbins
In interpreted mode, ordering by a UDT will result in an exception. For example:
{noformat}
import org.apache.spark.ml.linalg.{DenseVector, Vector}
val df = Seq.tabulate(30) { x =>
(x, x + 1, x + 2, new DenseVector(Array((x/100.0).toDouble, ((x +
1)/100.0).toDouble, ((x + 3)/100.0).toDouble)))
}.toDF("id", "c1", "c2", "c3")
df.createOrReplaceTempView("df")
// this works
sql("select * from df order by c3").collect
sql("set spark.sql.codegen.wholeStage=false")
sql("set spark.sql.codegen.factoryMode=NO_CODEGEN")
// this gets an error
sql("select * from df order by c3").collect
{noformat}
The second {{collect}} action results in the following exception:
{noformat}
org.apache.spark.SparkIllegalArgumentException: Type UninitializedPhysicalType
does not support ordered operations.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.orderedOperationUnsupportedByDataTypeError(QueryExecutionErrors.scala:348)
at
org.apache.spark.sql.catalyst.types.UninitializedPhysicalType$.ordering(PhysicalDataType.scala:332)
at
org.apache.spark.sql.catalyst.types.UninitializedPhysicalType$.ordering(PhysicalDataType.scala:329)
at
org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:60)
at
org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:39)
at
org.apache.spark.sql.execution.UnsafeExternalRowSorter$RowComparator.compare(UnsafeExternalRowSorter.java:254)
{noformat}
Note: You don't get an error if you use {{show}} rather than {{collect}}. This
is because {{show}} will implicitly add a {{limit}}, in which case the ordering
is performed by {{TakeOrderedAndProject}} rather than
{{UnsafeExternalRowSorter}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
