[
https://issues.apache.org/jira/browse/SPARK-59524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59524:
-----------------------------------
Labels: pull-request-available (was: )
> Enforce the array-key / HashPartitioner guard in subtractByKey and the Java
> fakeClassTag key path
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-59524
> URL: https://issues.apache.org/jira/browse/SPARK-59524
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Uroš Bojanić
> Priority: Minor
> Labels: pull-request-available
>
> The rule that rejects array keys under a HashPartitioner
> (SparkCoreErrors.hashPartitionerCannotPartitionArrayKeyError) is not enforced
> on two paths. These are pre-existing gaps (not regressions); they were
> surfaced during review of SPARK-59459 / apache/spark#58758, which
> consolidated the existing guard into a single helper.
> 1. RDD.subtractByKey never calls the guard, even though two of its overloads
> default to HashPartitioner. "arrPairs.subtractByKey(arrPairs)" silently
> returns every row, because SubtractedRDD matches Array keys by identity.
> 2. In the Java API, JavaSparkContext.parallelizePairs and
> JavaPairRDD.fromJavaRDD use fakeClassTag, so keyClass is always Object for
> such RDDs. The guard therefore never fires for byte[] keys in
> JavaPairRDD.partitionBy, join, or cogroup.
> Raised by Dongjoon Hyun in the #58758 review.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
