Kyoungha Min created BEAM-9743:
----------------------------------
Summary: TFRecordCodec not attempt to fully read header/footer
Key: BEAM-9743
URL: https://issues.apache.org/jira/browse/BEAM-9743
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Kyoungha Min
Assignee: Kyoungha Min
Seems like it only happens with Zstd compression (or any other picky input
stream that refuse to read fully). Zstd seems very picky at giving out data.
The parts with the issue are
[https://github.com/apache/beam/blob/c7911043510a266078a3dc8faef7a1dbe1f598c5/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L672]
[https://github.com/apache/beam/blob/c7911043510a266078a3dc8faef7a1dbe1f598c5/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L699]
And not so problem within the beam application, but still not following the
WritableByteChannel API,
[https://github.com/apache/beam/blob/c7911043510a266078a3dc8faef7a1dbe1f598c5/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L720-L727]
ReadableByteChannel/WritableByteChannel Javadoc specifies that they are not
required to read/write fully, and can refuse to read/write time to time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
