Hi Addison,
Sorry for the late reply.
I agree that the documentation can be significantly improved
and that adding compression could be a nice thing to have.
There is already a PR open for supporting writing SequenceFiles with
the StreamingFileSink. When this gets merged, you will be able to
Hi Addison,
I think it is a good idea to add some more details to the documentation.
Thus, it would be great if you could contribute how to enable compression.
Concerning the RollingPolicy, I've pulled in Klou who might give you more
details about the design decisions.
Cheers,
Till
On Wed, Nov
Just noticed one detail about using the BulkWriter interface, you no longer
can assign a rolling policy. That makes sense for formats like orc/parquet,
but perhaps not for simple text compression.
On Wed, Nov 14, 2018 at 1:43 PM Addison Higham wrote:
> HI all,
>
> I am moving some code to use
HI all,
I am moving some code to use the StreamingFileSink. Currently, it doesn't
look like there is any native support for compression (gzip or otherwise)
built into flink when using the StreamingFileSink. It seems like this is a
really common need that as far as I could tell, wasn't represented
