james-willis opened a new pull request, #3299: URL: https://github.com/apache/sedona/pull/3299
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3298 ## What changes were proposed in this PR? Two commits. **1. Bump GraphFrames 0.11.0 → 0.12.2** (`<graphframes.version>` in the root `pom.xml`). Sedona's only GraphFrames usage is `GraphFrame(...).connectedComponents` in `DBSCAN.scala`. The one breaking change in the 0.12.x line is the low-level Pregel edge-attribute contract (`requiredEdgeColumns`), which Sedona does not use. Artifact coordinates are unchanged and 0.12.2 is published for all three combinations Sedona builds against: `graphframes-spark3_2.12`, `graphframes-spark3_2.13`, `graphframes-spark4_2.13`. **2. Use `randomized_contraction` for DBSCAN's connected components.** ```scala val connectedComponentsDF = GraphFrame(corePointsDF, coreEdgesDf).connectedComponents .setAlgorithm("randomized_contraction") .setUseLabelsAsComponents(false) .run ``` Per the GraphFrames user guide, following graphframes/graphframes#888, `randomized_contraction` (Bögeholz, Brand & Todor, ICDE 2020) benchmarks better than the default `two_phase` with AQE and uses roughly half the memory. ### Behavior change `two_phase` returns the minimum original vertex ID per component when vertex IDs are integral; `randomized_contraction` always returns an arbitrary `Long`. For DBSCAN this is visible only when the input already has an integral `id` column — those cluster labels change from "smallest core-point id in the cluster" to an arbitrary `Long`. Without an `id` column, DBSCAN synthesizes one via `sha2(to_json(struct("*")), 256)`; those are non-integral, so `two_phase` already returns arbitrary `Long`s and nothing changes. Cluster labels remain `LongType`, the `-1` outlier sentinel is unaffected, and the `min(component)` border-point tie-break is unchanged. Callers treating the cluster column as an opaque grouping key are unaffected; callers relying on the numeric value of the label are not. ## How was this patch tested? Existing `GeoStatsSuite` DBSCAN coverage. Those tests join the SQL and function results on `id` and never assert specific cluster values, and the `useLabelsAsComponents` test asserts only that `cluster` is `LongType` — both still hold under randomized contraction. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
