[ 
https://issues.apache.org/jira/browse/SEDONA-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478920#comment-17478920
 ] 

Paweł Kociński commented on SEDONA-66:
--------------------------------------

[~magdalena.wojciak] Kind off, I was able to reproduce the exception. And based 
on my assumption zipPartitions function causes the issue with inconsistent 
results. Java version implementation lack of this function signature 

{ code:scala }

def zipPartitions[U, V](
other: JavaRDDLike[U, _],
f: FlatMapFunction2[JIterator[T], JIterator[U], V],
preservePartitioning: Boolean)

{ code }

I guess that preservePartitioning flag may help somehow but I didnt test it 
yet. 

 

What is actually in spark when zipPartitions is java called, it uses false then

{ code:scala }

def zipPartitions[B: ClassTag, V: ClassTag]
(rdd2: RDD[B])
(f: (Iterator[T], Iterator[B]) => Iterator[V]): RDD[V] = withScope {
zipPartitions(rdd2, preservesPartitioning = false)(f)
}

 

We may need sth like this in JavaRDDLike trait but I am not 100 percent sure 

{ code }

def zipPartitions[U, V](
other: JavaRDDLike[U, _],
f: FlatMapFunction2[JIterator[T], JIterator[U], V],
preservePartitioning: Boolean): JavaRDD[V] = {
def fn: (Iterator[T], Iterator[U]) => Iterator[V] = {
(x: Iterator[T], y: Iterator[U]) => f.call(x.asJava, y.asJava).asScala
}
JavaRDD.fromRDD(
rdd.zipPartitions(
other.rdd, preservePartitioning
)(fn)(other.classTag, fakeClassTag[V]))(fakeClassTag[V])
}

{ code:scala }

I am focusing on osm parser for Sedona (it takes more time than i expected and 
my resources are limited recently unfortunately)

You can try to build that and use in sedona and check if that resolves problem. 
I can do that during the closest weekend

 

 

> Coalesce after spatial join results in dropped rows.
> ----------------------------------------------------
>
>                 Key: SEDONA-66
>                 URL: https://issues.apache.org/jira/browse/SEDONA-66
>             Project: Apache Sedona
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: James Kyle
>            Assignee: Paweł Kociński
>            Priority: Major
>         Attachments: image-2021-09-30-11-41-43-840.png, 
> image-2021-09-30-11-43-18-164.png
>
>
> The code snippet we were using internal that reliably reproduced this error 
> looks something like
>  leftSpatialRDD.spatialJoinFlat(rightSpatialRDD, 
> ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS) 
> repartition does _not_ drop rows and serves as a workaround..
> Below is the summary of our internal conversation that potentially root 
> caused the problem. I've bolded the final sentence which is the summary of 
> the potential root cause. 
>  
>  ISSUE 1
>  
> {code:java}
>    val spatialRDD = makeSpatial(ds)
>     setSpatialPartitioner(spatialRDD)
>  
>     val joined = spatialRDD.spatialJoinFlat(spatialRDD, 
> ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS)
>     val joinedNew = spatialRDD.spatialJoinFlat(spatialRDD, ALLOW_OVERLAP)
>  
> {code}
>  
> zipPartitions, and within each partition, it does something like indexed/for 
> loop element checking
> !image-2021-09-30-11-41-43-840.png|width=819,height=454!
> then looking at the DAG of executed code, there are something interesting , 
> first is SparkSQL inject a Scan operator between Sedona's RDD operation and 
> the following DataFrame operations (.coalesce), and zipPartitions and 
> coalesce is put in the same stage, the colocated zipPartitions and coalesce 
> means that, even the joined RDD from Sedona is supposed to have N partitions, 
> it will still converge to X partitions where X is specified by coalesce,  
> !image-2021-09-30-11-43-18-164.png|width=575,height=972!
> assumption of self join, and the implementation of coalesce, selfjoin assumes 
> that the partitions will be 1:1 mapped, however, coalesce is a wrapper of 
> upstream iterators which doesn't guarantee the 1:1 mapped, *it is highly 
> possible that the highlight* *match in the screenshot returns a lot of false* 
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to