jiayuasu opened a new issue, #3128:
URL: https://github.com/apache/sedona/issues/3128
### Description
While building the `netcdf.metadata` data source (#3115, merged in #2829),
several defects were found and fixed in code patterns that `netcdf.metadata`
originally copied from the `geotiff.metadata` data source (#2846). The same
defects still exist in the `geotiff.metadata` package on master. The
corresponding fixes are already on master in
`spark/common/.../io/netcdfmetadata/`, so this is a targeted transplant.
### Defects to fix in `spark/common/.../io/geotiffmetadata/`
**Correctness (silent wrong results):**
1. **Wrong Hive partition values in bin-packed partitions.**
`GeoTiffMetadataPartitionReaderFactory` wraps the whole multi-file partition in
one `PartitionReaderWithPartitionValues` using
`partition.files.head.partitionValues`, so when files from different Hive
partitions (e.g. `year=2020/`, `year=2021/`) are packed into one Spark
partition, every row reports the first file's partition values. Fix: per-file
reader wrapping, as in `NetCdfMetadataConcatPartitionReader`.
2. **Scan equality ignores pushdown state.** `GeoTiffMetadataScan` is a case
class, but `FileScan` provides concrete `equals`/`hashCode`, which suppresses
the synthesized ones — `pushedLimit` and `options` do not participate in
equality. Since the scan alone enforces the pushed limit (`isPartiallyPushed =
false` removes the Limit operator), exchange/subquery reuse can serve a limited
scan's result for an unlimited one. Fix: override `equals` to include `options`
and `pushedLimit`, and derive `hashCode` from the same fields, as in
`NetCdfMetadataScan`.
**Crash:**
3. **`fallbackFileFormat` returns null.** `CREATE TABLE ... USING
geotiff.metadata` throws a `NullPointerException` (Spark instantiates the V1
fallback format for catalog paths). Fix: a stub `FileFormat` that fails on
instantiation with a descriptive `UnsupportedOperationException`, which also
rejects `CREATE TABLE` with an explicit schema (that path skips `inferSchema`),
as in `NetCdfMetadataUnsupportedFileFormat`.
**Behavior:**
4. **Directory handling overrides explicit options.** Single-directory loads
force `recursiveFileLookup=true`, overriding an explicit `false` and silently
disabling Hive-style partition discovery; multi-directory loads bypass the
`.tif`/`.tiff` extension filter entirely. Fix: apply both as defaults only, for
any directory root.
5. **Option presence checks are case-sensitive.** Defaults are guarded with
`containsKey` on a copied `HashMap` instead of the `CaseInsensitiveStringMap`,
so an explicit mixed-case option (e.g. `RecursiveFileLookup=false`) can
nondeterministically lose to the lowercase default.
6. **A glob path combined with an explicit `pathGlobFilter` drops one
constraint.** `.load("/dir/a*.tif")` with `pathGlobFilter="b*.tif"` rewrites
the path to `/dir` keeping only one filter; native Spark semantics apply both.
Fix: skip the glob rewrite when the user supplied a filter.
### Test plan
Port the corresponding regression tests from `netcdfMetadataTest` to the
GeoTIFF suite: Hive-partitioned directory with per-file partition values,
limit/exchange-reuse scan identity, `CREATE TABLE` (with and without explicit
schema) failing with a clear error, explicit `recursiveFileLookup=false`
keeping partition discovery, mixed-case option spelling, multi-directory
extension filtering, and glob + filter combination.
### Related
- #2846 (`geotiff.metadata` data source)
- #3115 / #2829 (`netcdf.metadata`, where these fixes landed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
