james-willis commented on issue #3298:
URL: https://github.com/apache/sedona/issues/3298#issuecomment-5460106889
Note for anyone finding this later: #3299 landed **only the dependency
bump** to GraphFrames 0.12.2. The second half of this issue — switching
DBSCAN's connected components to `randomized_contraction` — was **not** done.
This issue auto-closed because I left `Closes #3298` in #3299 after reducing
that PR's scope; the close is an artifact of the PR wording, not a signal that
the algorithm switch shipped. Leaving it closed since the bump did land, but
flagging it so the remaining work is not lost.
Still outstanding, if anyone wants to pick it up once a GraphFrames release
includes graphframes/graphframes#893:
1. `randomized_contraction` mishandled non-integral vertex IDs
(graphframes/graphframes#892), which is fatal here because `DBSCAN.dbscan`
assigns ids as `sha2(to_json(struct("*")), 256)`. Fixed upstream in #893,
merged to `main`, not yet released.
2. Spark 3.4 needs a fallback regardless: `randomized_contraction` uses
`functions.call_function`, added in Spark 3.5, and `graphframes-spark3_*` is
compiled against 3.5.x — so on a 3.4 runtime it throws `NoSuchMethodError`.
Sedona still builds and tests 3.4.0. This is independent of #892 and is not
fixed by #893.
3. Cluster labels change for inputs that already have an integral `id`
column: `two_phase` returns the minimum core-point id per cluster,
`randomized_contraction` an arbitrary `Long`. Inputs without an `id` column are
unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
