mattcasters opened a new pull request, #9114:
URL: https://github.com/apache/hop/pull/9114
## Summary
Spark file output on Windows `local[*]` still fails after #8951 when the
commit lists the output directory. Hadoop 3.4.2 `FileUtil.canRead` calls
`NativeIO.Windows.access0` on every Windows directory listing and does not
check that `hadoop.dll` loaded. `UnsatisfiedLinkError` is not an `IOException`,
so it escapes. The fallback from #8951 skipped chmod and owner lookups only.
A `file:///` path is how that write reaches the local file system. A typed
`file:///C:/...` URI, or a variable such as `${PROJECT_HOME}` stored as a VFS
file URI, resolves to the `file` scheme and is passed to Spark unchanged. It
uses `fs.file.impl`. Stripping the prefix still lists the directory through the
same local file system. The minimal Generate Rows pipeline never lists a
directory, which is why it succeeds while Spark File Output fails. `Reusing
default SparkSession` means this run kept the session the earlier pipeline
created. That session already had the fallback, and the stack still names
`RawLocalFileSystem.listStatus` because the method was inherited.
The Windows raw file system now lists with `java.io.File` when winutils and
the native library are both absent. Checksum files stay hidden by the checksum
wrapper. `hdfs://` and `s3a://` are unchanged.
Fixes #9113
## Test plan
- [x] `HopWindowsLocalFileSystemTest` (4) on Linux: decision table,
mkdir/create, `file:///` listing through the Java path (checksum file hidden,
missing path, single file), unreadable directory
- [x] Checkstyle on `hop-engines-spark` (0 violations) and Spotless
- [ ] Windows `local[*]` Spark File Output to a `file:///` directory with
`HADOOP_HOME` unset
------------------------
- [ ] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [x] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
- [x] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
