Eliaaazzz opened a new issue, #39336:
URL: https://github.com/apache/beam/issues/39336
### What happened?
On Windows, submitting any portable pipeline to a job server fails during
artifact staging, before the pipeline runs. The staged filename contains a
colon, which is not a legal path character on Windows.
Job server log:
```
java.util.concurrent.ExecutionException: java.nio.file.InvalidPathException:
Illegal char <:> at index 3:
1-0:ref_Environment_default-submission_environment_dependencies.txt
Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index 3:
1-0:ref_Environment_default-submission_environment_dependencies.txt
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:170)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247)
```
Reproduced on Windows 11, JDK 11, with the Spark job server
(`--spark-master-url local[4]`) and `--environment_type=LOOPBACK`, SDK and job
server both built from the same commit. The staging code is in
`runners/java-fn-execution` and is shared by the portable runners, so I would
expect this to affect the other portable runners too, though I only tried Spark.
### It is not specific to any pipeline shape
Both of these fail identically and produce no output:
- an unbounded `Read` (splittable DoFn)
- `Create([1, 2, 3]) | Map` , a bounded pipeline with no source and no SDF
Across five submissions the job server logged the same `Illegal char <:>`
ten times, so it is deterministic rather than intermittent.
### Where the colon comes from
`ArtifactStagingService.createFilename` builds the name out of the
environment id:
```java
return clip(
String.format("%s-%s-%s", idGenerator.getId(), clip(environment, 25),
base), 100);
```
For the observed name
`1-0:ref_Environment_default-submission_environment_dependencies.txt` the parts
are id `1`, `clip(environment, 25)` = `0:ref_Environment_default`, which is
exactly 25 characters, and base `submission_environment_dependencies.txt`. So
the colon comes from the environment id being embedded verbatim.
`getDestination` then resolves that name against the staging directory, and
`LocalResourceId` rejects it on Windows.
Sanitizing characters that are not legal in a filename before building the
name would fix it. Colon is the one that bites here, but `*`, `?`, `"`, `<`,
`>` and `|` are also illegal on Windows.
While reading that method, the splitter just above it also looks off:
```java
// Limit to the last contiguous alpha-numeric sequence. In particular, this
will exclude
// all path separators.
List<String> components =
Splitter.onPattern("[^A-Za-z-_.]]").splitToList(path);
```
The pattern has a trailing `]` outside the character class, so it matches a
disallowed character followed by a literal `]` rather than any disallowed
character. That does not cause this bug, since the staged name is already a
bare filename, but it means the comment's intent is not met for the other
artifact roles.
### Related
#21025 is the same area on Windows but a different failure: an illegal `*`
in `removeStagedArtifacts`, which happens during cleanup after the job reaches
DONE. In that report the pipeline still ran. This one fails staging, so the job
never starts.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Java SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
