[
https://issues.apache.org/jira/browse/SPARK-48275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-48275:
--------------------------------
Assignee: Mao Li
> array_sort: Improve documentation for default comparator's behavior for
> different types
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-48275
> URL: https://issues.apache.org/jira/browse/SPARK-48275
> Project: Spark
> Issue Type: Documentation
> Components: SQL
> Affects Versions: 3.5.0
> Environment: Running in Databricks on [Databricks Runtime
> 15.1|https://docs.databricks.com/en/release-notes/runtime/15.1.html].
> Reporter: Matt Braymer-Hayes
> Assignee: Mao Li
> Priority: Trivial
> Labels: pull-request-available
>
> h1. tl;dr
> It would be helpful for the documentation for array_sort() to include the
> default comparator behavior for different array element types, especially
> structs. It would also be helpful for the
> {{{\{DATATYPE_MISMATCH.INVALID_ORDERING_TYPE }}}} error to recommend using a
> custom comparator instead of the default comparator, especially when sorting
> on a complex type (e.g., a struct containing an unorderable field, like a
> map).
>
> ----
> h1. Background
> The default comparator for {{array_sort()}} for struct elements is to sort by
> every field in the struct in schema order (i.e., ORDER BY field1, field2,
> ..., fieldN). This requires every field to be orderable: if they aren't an
> error occurs.
>
> Here's a small example:
> {code:java}
> import pyspark.sql.functions as F
> import pyspark.sql.types as T
> schema = T.StructType([
> T.StructField(
> 'value',
> T.ArrayType(
> T.StructType([
> T.StructField('orderable', T.IntegerType(), True),
> T.StructField('unorderable', T.MapType(T.StringType(),
> T.StringType(), True), True), # remove this field and both commands below
> succeed
> ]),
> False
> ),
> False
> )
> ])
> df = spark.createDataFrame([], schema=schema)
> df.select(F.array_sort(df['value'])).collect(){code}
> Output:
> {code:java}
> [DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] Cannot resolve
> "(namedlambdavariable() < namedlambdavariable())" due to data type mismatch:
> The `<` does not support ordering on type "STRUCT<orderable: INT,
> unorderable: MAP<STRING, STRING>>". SQLSTATE: 42K09 {code}
>
> If the default comparator doesn't work for a user (e.g., they have an
> unorderable field like a map in their struct), array_sort() accepts a custom
> comparator, where users can order array elements however they like.
>
> Building on the previous example:
>
> {code:java}
> import pyspark.sql as psql
> def comparator(l: psql.Column, r: psql.Column) -> psql.Column:
> """Order structs l and r by order field.
> Rules:
> * Nulls are last
> * In ascending order
> """
> return (
> F.when(l['order'].isNull() & r['order'].isNull(), 0)
> .when(l['order'].isNull(), 1)
> .when(r['order'].isNull(), -1)
> .when(l['order'] < r['order'], -1)
> .when(l['order'] == r['order'], 0)
> .otherwise(1)
> )
> df.select(F.array_sort(df['value'], comparator)).collect(){code}
> This works as intended.
>
> ----
> h1. Ask
> The documentation for {{array_sort()}} should include information on the
> behavior of the default comparator for various datatypes. For the
> array-of-unorderable-structs example, it would be helpful to know that the
> default comparator for structs compares all fields in schema order (i.e.,
> {{{}ORDER BY field1, field2, ..., fieldN{}}}).
>
> Additionally, when users passes an unorderable type to array_sort() and uses
> the default comparator, the returned error should recommend the user use a
> custom comparator instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
