Patrick Lucas created BEAM-12857:
------------------------------------
Summary: Unable to write to GCS due to IndexOutOfBoundsException
in FileSystems
Key: BEAM-12857
URL: https://issues.apache.org/jira/browse/BEAM-12857
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Affects Versions: 2.32.0, 2.31.0
Environment: Beam 2.31.0/2.32.0, Java 11, GCP Dataflow
Reporter: Patrick Lucas
I have a simple batch job, running on Dataflow, that reads from a GCS bucket,
filters the data, and windows and writes the matching data back to a different
path in the same bucket.
The job seems to succeed in reading and filtering the data, as well as writing
temporary files to GCS, but appears to fail when trying to rename the temporary
files to their final destination.
The IndexOutOfBoundsException is thrown from
[FileSystems.java:429|https://github.com/apache/beam/blob/v2.32.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L429]
(in 2.32.0), when the code calls {{.get(0)}} on the list returned by a call to
{{MatchResult#metadata()}}.
The javadoc for
[{{MatchResult#metadata()}}|https://github.com/apache/beam/blob/v2.32.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/MatchResult.java#L75-L80]
says,
{code:java}
/**
* {@link Metadata} of matched files. Note that if {@link #status()} is
{@link Status#NOT_FOUND},
* this may either throw a {@link java.io.FileNotFoundException} or return an
empty list,
* depending on the {@link EmptyMatchTreatment} used in the {@link
FileSystems#match} call.
*/
{code}
So possibly GCS is not returning any metadata for the (missing) destination
object? That seems unlikely, as I would expect many others would have already
run into this, but I don't see how this could be caused by my user code.
I have tested this on 2.31.0 and 2.32.0 getting the same error, but it's worth
noting that the logic in FileSystems.java changed a decent amount recently in
[#15301|https://github.com/apache/beam/pull/15301], and maybe have an effect on
this, but I haven't been able to test it since I'm working in a closed
environment and can only easily use released versions of Beam. Once a version
containing this change is released, I will upgrade and try again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
