Uroš Bojanić created SPARK-59524:
------------------------------------
Summary: Enforce the array-key / HashPartitioner guard in
subtractByKey and the Java fakeClassTag key path
Key: SPARK-59524
URL: https://issues.apache.org/jira/browse/SPARK-59524
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.4.0
Reporter: Uroš Bojanić
The rule that rejects array keys under a HashPartitioner
(SparkCoreErrors.hashPartitionerCannotPartitionArrayKeyError) is not enforced
on two paths. These are pre-existing gaps (not regressions); they were surfaced
during review of SPARK-59459 / apache/spark#58758, which consolidated the
existing guard into a single helper.
1. RDD.subtractByKey never calls the guard, even though two of its overloads
default to HashPartitioner. "arrPairs.subtractByKey(arrPairs)" silently returns
every row, because SubtractedRDD matches Array keys by identity.
2. In the Java API, JavaSparkContext.parallelizePairs and
JavaPairRDD.fromJavaRDD use fakeClassTag, so keyClass is always Object for such
RDDs. The guard therefore never fires for byte[] keys in
JavaPairRDD.partitionBy, join, or cogroup.
Raised by Dongjoon Hyun in the #58758 review.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
