jiayuasu opened a new pull request, #3300: URL: https://github.com/apache/sedona/pull/3300
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3297 ## What changes were proposed in this PR? Spark 3.4 reached end of life in October 2024 with 3.4.4 as its final maintenance release, and the major cloud runtimes have since removed it. **Build** - Remove the `spark/spark-3.4` module and the `sedona-spark-3.4` profile. - Make 3.5 the default `spark.version` / `spark.compat.version`, and the profile that deploys the parent and common modules (`skip.deploy.common.modules`). - Drop `spark-3.5` from `enable-all-submodules`, since it is now the default module. **CI** - Remove the 3.4 rows from the java, python, r and example matrices. - The python 3.4 row was the only one carrying Python 3.8 and shapely 1 coverage, so that row moves to Spark 3.5 rather than being deleted. - The Spark Connect step's `matrix.spark >= '3.4.0'` guard is now always true and has been removed. - `docs.yml` reads `spark.version` from the POM, so it follows the new default automatically. **Python** - Raise the pyspark floor to `>=3.5.0`. - Remove the `pyspark.__version__ < "3.4"` skip markers from the raster tests, and the imports they orphaned. **Scala** - `PartitionedFile`'s constructor is identical from 3.5.0 onwards, so `StacPartitionReader` now builds one directly instead of reflectively choosing between the 7- and 8-argument forms. Verified against the published jars: 3.5.0, 3.5.8, 4.0.2 and 4.1.1 all expose `(InternalRow, SparkPath, long, long, String[], long, long, Map)`; only 3.4 had the 7-argument form. The private-constructor reflection in `createSparkPath` goes with it, superseded by the public `SparkPath.fromPathString`. - `SchemaMergeUtils.safeMerge` **keeps** its reflective fallback. `StructType.merge(StructType, Boolean)` does not exist in Spark 3.5.0 — it arrives in 3.5.1 — and 3.5.0 remains supported. Only the stale comments were corrected. - The `AbstractCatalog` aggregate-builder reflection also stays; it splits Spark 3.x from 4.x and is still live for 3.5. **Docs** - Compatibility tables, compile instructions, and Maven coordinate tabs (EN and ZH). - `publish.md` and `snapshot.md`: the release and snapshot build matrices still named Spark 3.3, dropped back in 1.8.0, and would have been entirely invalid after this change. **Also** - Remove `spark-version-converter.py`, a Spark 2/3 helper whose target paths (`sql/src/main/scala/...`) have not existed since the module layout changed. Nothing in the tree references it. ## How was this patch tested? Builds, all `BUILD SUCCESS`: - `-Dspark=3.5 -Dscala=2.12` (JDK 11) - `-Dspark=3.5 -Dscala=2.13` (JDK 11) - `-Dspark=4.0 -Dscala=2.13` (JDK 17) - `-Dspark=4.1 -Dscala=2.13` (JDK 17) - `mvn clean install -DskipTests` with no flags, confirming the default now resolves to `sedona-spark-3.5_2.12` Targeted tests for the changed `StacPartitionReader` path — `StacPartitionReaderTest`, `StacBatchTest`, `StacDataSourceTest` — on both `3.5/2.12` and `4.1/2.13`: 32 succeeded, 0 failed on each. The 4 canceled tests are the env-gated live STAC service tests (`STAC_PUBLIC_URL`, `STAC_AUTH_URL`), canceled identically on both. `pre-commit run --all-files` passes, and `mvn spotless:apply` produces no changes. ## Did this PR include necessary documentation updates? - Yes, I have updated the documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
