hansva commented on PR #9211:
URL: https://github.com/apache/hop/pull/9211#issuecomment-6600207150
Seems my initial review missed one thing
**[bug]** `java.io.File` collapses the `//` of a URI: `new
File("gs://bucket/out/data.parquet").getParent()` returns `gs:/bucket/out`
(`s3://bkt/x` gives `s3:/bkt`). That directory is then passed to
`FileIO.write().to(directory)` and `withTempDirectory(directory)`, so writing
to `gs://` or `s3://`, the usual target on Dataflow and Spark, fails or writes
to the wrong place. The tests only use local paths, so they don't catch it.
**Suggestion:** split the filename at the last `/` as a string, or use
`FileSystems.matchNewResource(filename, false)` with `getCurrentDirectory()` /
`getFilename()`, and add a test that writes to a `file://` URI. Related:
`createParent()` resolves the same path through Hop VFS on the machine that
submits the pipeline, so with *create parent folder* on it needs local GCS/S3
credentials, while object stores have no real folders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
