mattcasters opened a new issue, #9113:
URL: https://github.com/apache/hop/issues/9113
### Apache Hop version?
2.20.0-SNAPSHOT
### Java version?
21
### Operating system
Windows
### What happened?
Follow-up to #8946. The permission workaround in #8951 lets a pipeline with
no file output run on Windows `local[*]` when `HADOOP_HOME` and
`hadoop.home.dir` are unset. Spark File Input → Sample Rows → Spark File Output
to a local `file:///` directory still fails while the output commit lists that
directory:
```text
java.lang.UnsatisfiedLinkError: 'boolean
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)'
at
org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:817)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1415)
at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1620)
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:802)
at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:1020)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334)
at
org.apache.hop.spark.pipeline.handler.SparkFileIoSupport.writeDataset(SparkFileIoSupport.java:119)
```
The log reuses the Spark session from the earlier successful run (`Reusing
default SparkSession`). That session already has the Windows local file system
from #8951. The new stack is not `setPermission` / winutils. Hadoop 3.4.2
`FileUtil.canRead` calls `NativeIO.Windows.access` on every Windows directory
listing and does not check that the native library loaded.
`UnsatisfiedLinkError` is not an `IOException`, so the catch in `canRead` does
not hide it. `hadoop.dll` is absent when winutils is absent, so the commit
fails.
A `file:///` path is the local file system. A typed `file:///C:/...` URI, or
a variable such as `${PROJECT_HOME}` stored as a VFS file URI, resolves to that
scheme and is passed to Spark as-is. It selects `fs.file.impl`. It does not
select a different file system, and removing the prefix still lists the
directory through the same local file system.
### Steps to reproduce
1. Windows, Java 21, Hop 2.20.0-SNAPSHOT, Spark 4.1.3, `HADOOP_HOME` unset.
2. Native Spark run configuration, master `local[*]`.
3. Spark File Input → Sample Rows → Spark File Output.
4. Read a local CSV and write to a local `file:///` directory.
5. Execute the pipeline.
### Expected behavior
The write commits. Files keep the permissions of the Hop process.
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
