[
https://issues.apache.org/jira/browse/SEDONA-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478484#comment-17478484
]
Magdalena Wójciak commented on SEDONA-66:
-----------------------------------------
[~imbruced], are you working on that?
> Coalesce after spatial join results in dropped rows.
> ----------------------------------------------------
>
> Key: SEDONA-66
> URL: https://issues.apache.org/jira/browse/SEDONA-66
> Project: Apache Sedona
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: James Kyle
> Assignee: Paweł Kociński
> Priority: Major
> Attachments: image-2021-09-30-11-41-43-840.png,
> image-2021-09-30-11-43-18-164.png
>
>
> The code snippet we were using internal that reliably reproduced this error
> looks something like
> leftSpatialRDD.spatialJoinFlat(rightSpatialRDD,
> ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS)
> repartition does _not_ drop rows and serves as a workaround..
> Below is the summary of our internal conversation that potentially root
> caused the problem. I've bolded the final sentence which is the summary of
> the potential root cause.
>
> ISSUE 1
>
> {code:java}
> val spatialRDD = makeSpatial(ds)
> setSpatialPartitioner(spatialRDD)
>
> val joined = spatialRDD.spatialJoinFlat(spatialRDD,
> ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS)
> val joinedNew = spatialRDD.spatialJoinFlat(spatialRDD, ALLOW_OVERLAP)
>
> {code}
>
> zipPartitions, and within each partition, it does something like indexed/for
> loop element checking
> !image-2021-09-30-11-41-43-840.png|width=819,height=454!
> then looking at the DAG of executed code, there are something interesting ,
> first is SparkSQL inject a Scan operator between Sedona's RDD operation and
> the following DataFrame operations (.coalesce), and zipPartitions and
> coalesce is put in the same stage, the colocated zipPartitions and coalesce
> means that, even the joined RDD from Sedona is supposed to have N partitions,
> it will still converge to X partitions where X is specified by coalesce,
> !image-2021-09-30-11-43-18-164.png|width=575,height=972!
> assumption of self join, and the implementation of coalesce, selfjoin assumes
> that the partitions will be 1:1 mapped, however, coalesce is a wrapper of
> upstream iterators which doesn't guarantee the 1:1 mapped, *it is highly
> possible that the highlight* *match in the screenshot returns a lot of false*
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)