jiayuasu opened a new pull request, #3142: URL: https://github.com/apache/sedona/pull/3142
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #3131 ## What changes were proposed in this PR? `geotiff.metadata` and `netcdf.metadata` rewrote a single file-glob path such as `/dir/a*.nc` into path `/dir` plus `pathGlobFilter="a*.nc"`. Because `pathGlobFilter` matches leaf file names at any depth, combining the glob with an explicit `recursiveFileLookup=true` widened the match into nested files such as `/dir/sub/a1.nc` — unlike the native Spark glob, which matches direct children only. Directory globs such as `/data/region*` got neither of the directory-scan defaults: the `getFileStatus` probe fails on a glob string, so the expanded directories were listed non-recursively and unfiltered, inconsistent with loading the same directories by explicit path. This PR takes the second direction proposed in #3131: - **The glob-to-filter rewrite is removed.** File paths and file globs reach Spark untouched and keep native listing semantics in every option combination — the explicit-`pathGlobFilter` case has taken this path since #3129 and passes CI across the Spark matrix, so the rewrite no longer earned its divergence. As in every other Spark file source, a file glob that matches nothing is now a `PATH_NOT_FOUND` error rather than a silently empty result. - **Directory detection goes through `FileSystem.globStatus`,** which resolves plain paths and glob patterns alike: a path stands for directories when it matches at least one status and all matches are directories. Directory globs therefore receive the same recursive-scan and extension-filter defaults as explicit directory paths. The defaults still apply only when every root resolves to directories, preserving the mixed-load safeguard from #3129. Both data sources are changed identically and stay aligned. ## How was this patch tested? Six new regression tests, three per data source, mirrored across `geotiffMetadataTest` and `netcdfMetadataTest`: - a file glob with explicit `recursiveFileLookup=true` matches direct children only (nested `a2.nc`/`a2.tiff` is not pulled in), - a directory glob receives the directory-scan defaults (a nested file is found via recursion; a non-raster decoy is excluded by the extension filter), - a file glob matching nothing raises `AnalysisException` (`PATH_NOT_FOUND`). Full suites pass: 86/86 across both (`-Dspark=3.5 -Dscala=2.12`), including the pre-existing glob, directory, multi-directory, partition-discovery, and option-precedence tests. ## Did this PR include necessary documentation updates? - Yes, I have updated the documentation. Both data source pages now document the native glob semantics (direct children only, missing-path error for empty globs) and that directory globs receive the directory-scan defaults. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
