GitHub user dalegaard opened a pull request:
https://github.com/apache/flink/pull/1826
Refactor rolling sink writer
Implements FLINK-3637 by changing the Writer interface such that the Writer
must handle the output stream, instead of having the RollingSink handling it.
This makes it possible to write outputs for ORC and Parquet.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dalegaard/flink refactor_rolling_sink_writer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1826.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1826
----
commit af7ac4f7ff1713951f2e9e3465761aafd26cf73a
Author: Lasse Dalegaard <[email protected]>
Date: 2016-03-22T09:24:11Z
[FLINK-3637] Refactored output stream handling from RollingSink.
The Writer interface now deals directly with filesystem and path, rather
than the raw output stream.
Since the RollingSink no longer has access to the raw output stream, it
cannot directly determine the current size of the file. A getPos()
method has been added to the Writer interface, so the RollingSink can.
retrieve the current file size.
Finally, flush() has been extended to return the offset that the file
must be truncated to at recovery.
commit 0a1373a21b3f6275ca53b28602453f5fd5e81a64
Author: Lasse Dalegaard <[email protected]>
Date: 2016-03-22T09:24:43Z
[FLINK-3637] Updated writer implementations for new interface.
SequenceFileWriter and StringWriter are both simple outputs that work
directly on a file in HDFS, using the logic that used to reside in
RollingSink. This logic has been moved into a new class,
SimpleBaseWriter, which both StringWriter and SequenceFileWriter extend.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
