[
https://issues.apache.org/jira/browse/SEDONA-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479237#comment-17479237
]
Magdalena Wójciak commented on SEDONA-66:
-----------------------------------------
[~imbruced] if you've already worked on it, I can start working on another
issue. I'd like to try to solve:
[SEDONA-65|https://issues.apache.org/jira/browse/SEDONA-65?jql=project%20%3D%20SEDONA%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20updated%20DESC%2C%20affectedVersion%20ASC%2C%20cf%5B10010%5D%20ASC%2C%20priority%20DESC]
> Coalesce after spatial join results in dropped rows.
> ----------------------------------------------------
>
> Key: SEDONA-66
> URL: https://issues.apache.org/jira/browse/SEDONA-66
> Project: Apache Sedona
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: James Kyle
> Assignee: Paweł Kociński
> Priority: Major
> Attachments: image-2021-09-30-11-41-43-840.png,
> image-2021-09-30-11-43-18-164.png
>
>
> The code snippet we were using internal that reliably reproduced this error
> looks something like
> leftSpatialRDD.spatialJoinFlat(rightSpatialRDD,
> ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS)
> repartition does _not_ drop rows and serves as a workaround..
> Below is the summary of our internal conversation that potentially root
> caused the problem. I've bolded the final sentence which is the summary of
> the potential root cause.
>
> ISSUE 1
>
> {code:java}
> val spatialRDD = makeSpatial(ds)
> setSpatialPartitioner(spatialRDD)
>
> val joined = spatialRDD.spatialJoinFlat(spatialRDD,
> ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS)
> val joinedNew = spatialRDD.spatialJoinFlat(spatialRDD, ALLOW_OVERLAP)
>
> {code}
>
> zipPartitions, and within each partition, it does something like indexed/for
> loop element checking
> !image-2021-09-30-11-41-43-840.png|width=819,height=454!
> then looking at the DAG of executed code, there are something interesting ,
> first is SparkSQL inject a Scan operator between Sedona's RDD operation and
> the following DataFrame operations (.coalesce), and zipPartitions and
> coalesce is put in the same stage, the colocated zipPartitions and coalesce
> means that, even the joined RDD from Sedona is supposed to have N partitions,
> it will still converge to X partitions where X is specified by coalesce,
> !image-2021-09-30-11-43-18-164.png|width=575,height=972!
> assumption of self join, and the implementation of coalesce, selfjoin assumes
> that the partitions will be 1:1 mapped, however, coalesce is a wrapper of
> upstream iterators which doesn't guarantee the 1:1 mapped, *it is highly
> possible that the highlight* *match in the screenshot returns a lot of false*
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)