[
https://issues.apache.org/jira/browse/FLINK-11395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997078#comment-16997078
]
Grzegorz Kołakowski edited comment on FLINK-11395 at 12/16/19 3:18 PM:
-----------------------------------------------------------------------
What about replacing {{Encoder}} interface with something similar to
{{BulkWriter}} and a factory with, for instance, {{createForResume}} and
{{createForNew}} methods? This way we will be able to easily provide adapters
for avro specific writers: {{org.apache.avro.file.DataFileWriter}},
{{org.apache.avro.generic.GenericDatumWriter}}. Current implementation of
{{Encoder}} makes it impossible to recognize whether we are writing to a new
file or a reopened one.
was (Author: grzegorz_kolakowski):
What about replacing {{Encoder}} interface with something similar to
{{BulkWriter}} and a factory with, for instance, {{createForResume}} and
{{createForNew}} methods? This way we will be able to easily provide adapters
for avro specific writers: {{org.apache.avro.file.DataFileWriter}},
{{org.apache.avro.generic.GenericDatumWriter}}. Current implementation of
{{Encoder}} makes it impossible to recognize whether we are writing to a new
file or reopened one.
> Support for Avro StreamingFileSink
> ----------------------------------
>
> Key: FLINK-11395
> URL: https://issues.apache.org/jira/browse/FLINK-11395
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / FileSystem
> Reporter: Elango Ganesan
> Assignee: vinoyang
> Priority: Major
>
> Current implementation for StreamingFileSink supports Rowformat for text and
> json files . BulkWriter has support for Parquet files . Out of the box
> RowFormat does not seem to be right fit for writing Avro files as avro files
> need to persist metadata when opening new file. We are thinking of
> implementing Avro writers similar to how Parquet is implemented . I am happy
> to submit a PR if this approach sounds good .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
