jiayuasu opened a new pull request, #3129: URL: https://github.com/apache/sedona/pull/3129
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3128 ## What changes were proposed in this PR? Apply to `geotiff.metadata` the fixes that hardened the same code patterns in `netcdf.metadata` during the #2829 reviews. `netcdf.metadata` was originally modeled on `geotiff.metadata`, so these defects exist identically in the GeoTIFF package on master. ### Correctness (silent wrong results) 1. **Per-file partition values.** A bin-packed partition wrapped one reader in a single `PartitionReaderWithPartitionValues` using `partition.files.head.partitionValues`, so files from different Hive partitions (e.g. `year=2020/`, `year=2021/`) all reported the first file's partition values. Each file's reader is now wrapped with its own values (`GeoTiffMetadataConcatPartitionReader`; sub-readers open lazily, at most one file open at a time). 2. **Scan equality and hashing include pushdown state.** `FileScan`'s concrete `equals`/`hashCode` suppress the case-class-synthesized versions, so `pushedLimit` and `options` did not participate in equality. Since the scan alone enforces the pushed limit (`isPartiallyPushed = false` removes the Limit operator), exchange/subquery reuse could serve a limited scan's result for an unlimited one. `equals` now includes both fields and `hashCode` is derived from every equality field. ### Crash 3. **`fallbackFileFormat` no longer returns null.** `CREATE TABLE ... USING geotiff.metadata` threw a `NullPointerException`. The fallback is now a stub `FileFormat` that fails on instantiation with a descriptive `UnsupportedOperationException`, which also rejects `CREATE TABLE` with an explicit schema (that path skips `inferSchema`). ### Behavior 4. **Directory defaults never override explicit options.** Single-directory loads forced `recursiveFileLookup=true`, overriding an explicit `false` and silently disabling Hive-style partition discovery; multi-directory loads bypassed the GeoTIFF extension filter entirely. Both are now defaults only, applied for any directory root. 5. **Option presence checks are case-insensitive.** Defaults were guarded with `containsKey` on a copied `HashMap`, so an explicit mixed-case option (e.g. `RecursiveFileLookup=false`) could nondeterministically lose to the lowercase default. 6. **A glob path combined with an explicit `pathGlobFilter` keeps both constraints.** `.load("/dir/a*.tif")` with `pathGlobFilter="b*.tif"` previously rewrote the path and kept only one filter; the rewrite is now skipped when the user supplied a filter, so Spark applies both natively. Every fix mirrors its review-hardened counterpart in `spark/common/.../io/netcdfmetadata/`, adjusted only for naming and the `.tif`/`.tiff` extensions. Documentation gains the directory-load defaults and the unsupported-catalog-operations note, mirroring the NetCDF page. ## How was this patch tested? 7 new tests in `geotiffMetadataTest.scala` (19 total), mirroring the netcdf.metadata regression coverage: - Hive-style partition discovery with an explicit mixed-case `RecursiveFileLookup=false`, asserting every row carries its own file's partition value even when bin-packed into one Spark partition - Scan hash/equality consistency across equal, limit-differing, and options-differing copies - Glob path + explicit `pathGlobFilter` returning the native empty intersection (with a glob-only sanity check) - Multi-directory loads filtering non-GeoTIFF files - `CREATE TABLE ... USING geotiff.metadata` failing with a clear error instead of an NPE, both without and with an explicit schema Full `geotiffMetadataTest`, `netcdfMetadataTest`, and `rasteralgebraTest` suites pass on Spark 3.4/Scala 2.12 (240 tests); sources cross-compile on Spark 4.0/Scala 2.13. ## Did this PR include necessary documentation updates? - Yes, I have updated the documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
