Joel Robin created SPARK-61081:
----------------------------------
Summary: GROUP BY, DISTINCT and hash functions on GEOMETRY or
GEOGRAPHY fail with scala.MatchError
Key: SPARK-61081
URL: https://issues.apache.org/jira/browse/SPARK-61081
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Joel Robin
h3. Summary
A query fails with scala.MatchError or INTERNAL_ERROR when it groups,
deduplicates or hashes a GEOMETRY or GEOGRAPHY value. The analyzer accepts
these queries. The failure occurs at execution or during optimization.
h3. Environment
* Repository: https://github.com/apache/spark
* Branch: master
* Commit: 352ac9bf228 (2026-11-06), version 5.0.0-SNAPSHOT
* Configuration: default. spark.sql.geospatial.enabled is true by default.
* Run mode: local[4], Java 21
h3. Steps to reproduce
Run these statements in a Spark SQL session:
{code:sql}
-- 1. GROUP BY
SELECT g, COUNT(*) AS n
FROM (SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040',
4326) AS g FROM range(10))
GROUP BY g;
-- 2. COUNT(DISTINCT)
SELECT COUNT(DISTINCT g) AS cd
FROM (SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040',
4326) AS g FROM range(10));
-- 3. hash()
SELECT hash(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040',
4326)) AS h;
{code}
h3. Expected result
Each statement returns a correct result, or the analyzer stops it with a clear
error.
For comparison, ORDER BY g and g = g fail with a clear error:
DATATYPE_MISMATCH.INVALID_ORDERING_TYPE.
h3. Actual result
Statements 1 and 2 abort the job:
{noformat}
org.apache.spark.SparkException: Job aborted due to stage failure: ...
scala.MatchError: org.apache.spark.unsafe.types.BinaryView@15dc1861 (of class
org.apache.spark.unsafe.types.BinaryView)
at
org.apache.spark.sql.catalyst.expressions.InterpretedHashFunction.hash(hash.scala:713)
at
org.apache.spark.sql.catalyst.expressions.CollationAwareMurmur3Hash.computeHash(hash.scala:899)
at
org.apache.spark.sql.catalyst.expressions.HashExpression.eval(hash.scala:405)
at org.apache.spark.sql.catalyst.expressions.Pmod.eval(arithmetic.scala:1200)
at
org.apache.spark.sql.catalyst.expressions.InterpretedUnsafeProjection.apply(InterpretedUnsafeProjection.scala:84)
at
org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$8(ShuffleExchangeExec.scala:445)
{noformat}
Statement 3 fails during optimization:
{noformat}
[INTERNAL_ERROR] The Spark SQL phase optimization failed with an internal
error. ...
Caused by: scala.MatchError: org.apache.spark.unsafe.types.BinaryView@15dc1861
(of class org.apache.spark.unsafe.types.BinaryView)
at
org.apache.spark.sql.catalyst.expressions.InterpretedHashFunction.hash(hash.scala:713)
at
org.apache.spark.sql.catalyst.expressions.Murmur3Hash.computeHash(hash.scala:862)
at
org.apache.spark.sql.catalyst.expressions.HashExpression.eval(hash.scala:405)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.tryFold(expressions.scala:73)
{noformat}
These queries fail in the same way:
* approx_count_distinct(g)
* SELECT DISTINCT g
* DISTRIBUTE BY g, when g is not a constant. (With a constant,
OptimizeRepartition changes the query to one partition, and the query does not
fail.)
* xxhash64(g)
* GROUP BY and COUNT(DISTINCT) on GEOGRAPHY, for example ST_GeogFromWKB(...)
h3. Cause
# InterpretedHashFunction.hash (hash.scala:713) has no case for BinaryView.
BinaryView is the physical value of GEOMETRY and GEOGRAPHY.
# HashExpression.computeHashWithTailRec (hash.scala:638) has no case for
GeometryType or GeographyType. Code generation fails first, then Spark falls
back to the interpreted path, which also fails. With
spark.sql.codegen.factoryMode=CODEGEN_ONLY, the error is: scala.MatchError:
GeometryType(4326).
# The analyzer does not stop these queries. ExprUtils.checkValidGroupingExprs
(ExprUtils.scala:228-235) rejects only VariantType.
HashExpression.checkInputDataTypes rejects only MapType and VariantType.
h3. Related problem: spill to sort-based aggregation
A GROUP BY on a geometry key also fails when the hash aggregate spills, also
with one partition. GenerateOrdering generates c1.compare(c2) for the geometry
key, but BinaryView has only compareTo. To force the spill, set
spark.sql.TungstenAggregate.testFallbackStartsAt=1,2 and run:
{code:sql}
SELECT g, COUNT(*) AS n
FROM (SELECT /*+ REPARTITION(1) */
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS g FROM
range(10))
GROUP BY g;
{code}
Result:
{noformat}
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 30,
Column 20: A method named "compare" is not declared in any enclosing class nor
any supertype, nor through a static import
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:192)
at
org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:84)
at
org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:277)
at
org.apache.spark.sql.execution.aggregate.HashAggregateExec$.spillHashMapToSorter(HashAggregateExec.scala:1452)
{noformat}
h3. Note on existing tests
CollationAggregationSuite (SPARK-58847) expects GROUP BY key, geometry to
return a correct result. The test calls repartition(1) first. With one
partition, the plan has no hash shuffle, so the test does not find this
problem. The same query without repartition(1) fails with the MatchError above.
h3. Workaround
Group by the binary value: GROUP BY ST_AsBinary(g).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
