[
https://issues.apache.org/jira/browse/FLINK-13850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916054#comment-16916054
]
Gyula Fora commented on FLINK-13850:
------------------------------------
I think having clear and well structured code for configuration is very
important for these sinks. I don't think we have done a super good job at this
in the past.
We have already introduced 2 distinct methods to configure how the part files
are named, that pollute the existing builder code (all the constructors, fields
etc.)
As the whole StreamFileSink is about the part files I can easily imagine a few
other features users might want to add in the future.
* Pattern for naming the whole file, not only prefix, suffix
* Different folder for storing in-progress/pending files
Having a central place to configure part file creation related settings adds a
nice hierarchy to the configuration and makes it easier to use in real-life.
The PartFileConfig created in this PR is easily extensible with new options
that the StreamingFileSink can easily pick up in the future without changing
the format builder. The whole idea for the PartFileConfigBuilder is so that you
only define what you need and not everything (but it is extensible).
> Refactor part file configuration into a single method
> -----------------------------------------------------
>
> Key: FLINK-13850
> URL: https://issues.apache.org/jira/browse/FLINK-13850
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / FileSystem
> Reporter: Gyula Fora
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently there is only two methods on both format builders
> withPartFilePrefix and withPartFileSuffix for configuring the part files but
> in the future it is likely to grow.
> * More settings, different directories for pending / inprogress files etc
> I suggest we remove these two methods and replace them with a single :
> withPartFileConfig(..) where we use an extensible config class.
> This should be fixed before 1.10 in order to not release the other methods.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
