[
https://issues.apache.org/jira/browse/SEDONA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721001#comment-17721001
]
Kristin Cowalcijk commented on SEDONA-278:
------------------------------------------
The internal representation of GeometryUDT has changed since Apache Sedona
1.4.0. GeometryUDT columns saved in Delta tables or Parquet files by old
versions of Sedona cannot be recognized by newer versions.
You can serialize the geometry column in Delta lake table to WKB and save
another copy using Sedona 1.1.0, then {{st_geomfromwkb(geom_wkb)}} will
consistently work on all Sedona versions.
> WKB geometry column in Delta Lake table not recognized as such in functions
> ST_GeomFromWKB or ST_AsText
> -------------------------------------------------------------------------------------------------------
>
> Key: SEDONA-278
> URL: https://issues.apache.org/jira/browse/SEDONA-278
> Project: Apache Sedona
> Issue Type: Bug
> Affects Versions: 1.4.0
> Environment: Databricks Runtime 12.1 with Apache Sedona 1.4.0
> Reporter: Quentin Mailhos
> Priority: Major
> Labels: Binary, WKB
> Fix For: 1.4.1
>
> Attachments: MicrosoftTeams-image (3).png
>
>
> After upgrading to Databricks DBR 12.1 with Apache Sedona 1.4.0, functions
> ST_GeomFromWKB and ST_AsText fail to read a Well-Known Binary (WKB) type
> column from a Delta Lake table, please see attached screenshot.
> SQL error message is ambiguous:
> [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve
> "st_geomfromwkb(geom)" due to data type mismatch: parameter 1 requires
> ("STRING" or "BINARY") type, however, "geom" is of "BINARY" type.;
> Spark error message as follows:
> Caused by: org.apache.spark.sql.AnalysisException: Invalid Spark read type:
> expected optional group geom (LIST) \{ repeated group list { required int32
> element (INTEGER(8,true)); } } to be list type but found Some(BinaryType)
> Workflow used to work just fine in Databricks DBR 9 LTS with Apache Sedona
> 1.1.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
