james-willis opened a new issue, #3298:
URL: https://github.com/apache/sedona/issues/3298
Sedona pins GraphFrames 0.11.0 via `<graphframes.version>` in the root
`pom.xml`. GraphFrames has shipped 0.12.0, 0.12.1 and 0.12.2 since. Two things
in those releases matter to Sedona.
## 1. The bump
Sedona's only GraphFrames usage is `GraphFrame(...).connectedComponents` in
`spark/common/src/main/scala/org/apache/sedona/stats/clustering/DBSCAN.scala`.
The single breaking change across the 0.12.x line is in the low-level Pregel
API — edge attributes are no longer implicitly packed into a persisted
`StructType` and must be requested explicitly via `requiredEdgeColumns` — which
Sedona does not use.
Artifact coordinates are unchanged, and 0.12.2 is published for all three
combinations Sedona builds against: `graphframes-spark3_2.12`,
`graphframes-spark3_2.13`, `graphframes-spark4_2.13`.
## 2. Use `randomized_contraction` for DBSCAN's connected components
GraphFrames offers three connected-components implementations: `graphx`,
`two_phase` (the default), and `randomized_contraction`, the last based on
Bögeholz, Brand and Todor, *"In-database connected component analysis"* (ICDE
2020). Following the performance work in graphframes/graphframes#888, the
GraphFrames user guide now states that `randomized_contraction` performs better
on benchmarks than `two_phase` with AQE and needs roughly half the memory, and
that it is not the library default only for backwards-compatibility reasons.
DBSCAN is a good fit for it, because Sedona does not expose GraphFrames'
component IDs as stable identifiers — they are opaque cluster labels.
### Behavior change
`two_phase` returns the minimum original vertex ID within each component
when vertex IDs are integral; `randomized_contraction` always returns an
arbitrary `Long`.
For DBSCAN this is only visible when the input already has an `id` column of
integral type — those cluster labels change from "smallest core-point id in the
cluster" to an arbitrary `Long`. When the input has no `id` column, DBSCAN
synthesizes one with `sha2(to_json(struct("*")), 256)`; those IDs are
non-integral, so `two_phase` already returns arbitrary `Long`s today and
nothing changes.
Cluster labels remain `LongType`, the `-1` outlier sentinel is unaffected,
and the `min(component)` tie-break used to assign border points to a cluster is
unchanged. Callers that treat the cluster column as an opaque grouping key —
which is how it is documented — are unaffected. Callers that relied on the
numeric value of the label are not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
