Thanks a lot for the suggestion Till!
I ended up using your suggestion of extending StreamWriterBase and wrapping the
FSDataOutputStream with GZIPOutputStream.
On 2018/03/28 09:44:26, Till Rohrmann <trohrm...@apache.org> wrote:
> Hi,
>
> the SequenceFileWriter and the AvroKeyValueSinkWriter both support
> compressed outputs. Apart from that, I'm not aware of any other Writers
> which support compression. Maybe you could use these two Writers as a
> guiding example. Alternatively, you could try to extend the
> StreamWriterBase and wrapping the outStream into a GZIPOutputStream.
>
> Cheers,
> Till
>
> On Wed, Mar 28, 2018 at 1:59 AM, l...@lyft.com <l...@lyft.com> wrote:
>
> > I want to upload a compressed file (gzip preferrably) using the Bucketing
> > Sink. What is the best way to do this? Would I have to implement my own
> > Writer that does the compression? Has anyone done something similar?
> >
>
