james-willis commented on issue #3298:
URL: https://github.com/apache/sedona/issues/3298#issuecomment-5458942102
Status update: this is now blocked only on a GraphFrames release, not on a
defect.
The switch to `randomized_contraction` was dropped from #3299 because the
algorithm could not run against Sedona's DBSCAN at all — `DBSCAN.dbscan`
assigns vertex ids as `sha2(to_json(struct("*")), 256)`, and
`randomized_contraction` mishandled non-integral vertex IDs
(graphframes/graphframes#892).
That has since been fixed upstream in graphframes/graphframes#893, now
merged to `main`. Once a GraphFrames release includes it, the switch becomes
viable again.
Two things to carry over when it is revisited:
1. **Spark 3.4 needs a fallback.** `randomized_contraction` is implemented
with `functions.call_function`, which was only added in Spark 3.5, and the
published `graphframes-spark3_*` artifact is compiled against 3.5.x. On a Spark
3.4 runtime the call links but throws `NoSuchMethodError` at execution time.
Sedona still builds and tests Spark 3.4.0, so the default needs to be gated on
the runtime Spark version. This is independent of #892 and is not fixed by #893.
2. **Cluster labels change for integral-id inputs.** `two_phase` returns the
minimum core-point id per cluster; `randomized_contraction` returns an
arbitrary `Long`. Inputs without an `id` column are unaffected, since the
synthesized sha2 id is non-integral and `two_phase` already randomizes there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
