[
https://issues.apache.org/jira/browse/FLINK-10469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636085#comment-16636085
]
ASF GitHub Bot commented on FLINK-10469:
----------------------------------------
NicoK opened a new pull request #6788: [FLINK-10469][core] make sure to always
write the whole buffer to FileChannel
URL: https://github.com/apache/flink/pull/6788
## What is the purpose of the change
Currently all the calls to one of the `FileChannel.write()` methods assume
that this method will not return before the whole buffer is written, like the
one in `AsynchronousFileIOChannel.write()`. However, this assumption may not be
right for all the environments and there is no such guarantee.
## Brief change log
- add `FileUtils#writeCompletely()` that loops through the provided buffer
until it is completely written
- adapt all uses of `FileChannel.write()` to use this helper method
## Verifying this change
No tests were added.
- Does it make sense to test the utility method? no, the functionality is
too basic
- Previously existing tests for writing to a `FileChannel` apparently only
covered complete writes through a single call (no failed tests afaik).
Extending them may be difficult difficult (`FileChannel` usually internal, need
to override it to force the behaviour) and potentially not necessary either -
waiting for a second opinion here.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no** (per
buffer written to disk)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not documented**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> FileChannel may not write the whole buffer in a single call to
> FileChannel.write(Buffer buffer)
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-10469
> URL: https://issues.apache.org/jira/browse/FLINK-10469
> Project: Flink
> Issue Type: Bug
> Components: Core, Network
> Affects Versions: 1.4.1, 1.4.2, 1.5.3, 1.6.0, 1.6.1, 1.7.0, 1.5.4, 1.6.2
> Reporter: Yun Gao
> Assignee: Yun Gao
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0, 1.6.2, 1.5.5
>
>
> Currently all the calls to _FileChannel.write(ByteBuffer src)_ assumes that
> this method will not return before the whole buffer is written, like the one
> in _AsynchronousFileIOChannel.write()._
>
> However, this assumption may not be right for all the environments. We have
> encountered the case that only part of a buffer was written on a cluster with
> a high IO load, and the target file got messy.
>
> To fix this issue, I think we should add a utility method in the
> org.apache.flink.util.IOUtils to ensure the whole buffer is written with a
> loop，and replace all the calls to _FileChannel.write(ByteBuffer)_ with this
> new method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
