jiayuasu opened a new issue, #3131:
URL: https://github.com/apache/sedona/issues/3131
### Description
Two pre-existing path-handling limitations in the metadata data sources
(`geotiff.metadata`, `netcdf.metadata` — they share this logic) were reproduced
during the #3129 review. Neither was introduced by #3129 or #2829; both trace
back to the original `createTable` path handling in #2846.
### 1. The glob-to-filter rewrite widens matches into nested files under
recursive lookup
A single glob path such as `.load("/dir/a*.tif")` is internally rewritten to
path `/dir` with `pathGlobFilter="a*.tif"`. `pathGlobFilter` matches leaf file
names at any depth during listing, so combining the glob path with an explicit
`recursiveFileLookup=true` also matches nested files such as `/dir/sub/a1.tif`
— whereas the native Spark glob only matches direct children of `/dir`.
Possible directions:
- Skip the rewrite when the user set `recursiveFileLookup` explicitly
(mirroring how an explicit `pathGlobFilter` already disables it since #3129), or
- Evaluate whether the rewrite is still needed at all: the explicit-filter
path already defers to native Spark glob handling, so if native listing handles
`*.tif` globs correctly on current Spark versions, the rewrite could be removed
and every combination would get native semantics.
### 2. Directory globs do not receive the directory-scan defaults
A glob that matches directories, e.g. `.load("/data/region*/")`, gets
neither `recursiveFileLookup=true` nor the extension `pathGlobFilter`: the
directory probe (`FileSystem.getFileStatus`) fails on a glob string, and the
file-glob rewrite pattern only matches names ending in the raster extensions.
The expanded directories are therefore listed non-recursively and unfiltered —
inconsistent with loading the same directories by explicit path, where both
defaults apply.
A fix likely requires glob-aware directory detection (e.g. expanding the
glob before probing, or treating a trailing `/` after a glob segment as a
directory indicator).
### Scope
Both limitations exist identically in
`spark/common/.../io/geotiffmetadata/GeoTiffMetadataDataSource.scala` and
`spark/common/.../io/netcdfmetadata/NetCdfMetadataDataSource.scala`; a fix
should cover both and keep their behavior aligned.
### Related
- #2846 (`geotiff.metadata`, origin of the path handling)
- #3115 / #2829 (`netcdf.metadata`)
- #3128 / #3129 (option-handling fixes; these two cases were explicitly
scoped out during review)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
