[
https://issues.apache.org/jira/browse/FLINK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17430423#comment-17430423
]
Fabian Paul commented on FLINK-24493:
-------------------------------------
[~rionmonster]
You have raised some good points let me try to answer some of them which
hopefully eases the development.
{quote}
I was curious if there was any documentation / examples / tests that I could
use to glean from with regards to the new Sink interface. I'm not terribly
familiar with it as much as the previous common uses of SinkFunctions, etc.
Could anyone point me in a direction on that front?
{quote}
I think a good way to start is to look at the original FLIP documentation
https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
and maybe take a look at the code of the KafkaSink which we developed recently.
{quote}
Given the changes made to the Sink interface, would the expectation be that
this new DemultiplexingSink inherits from a RichSinkFunction similar to my
previous implementation for the use-case for Elasticsearch? Or would it just
directly implement the Sink interface and the router would govern how to
construct the sink itself similar to the port of the Elasticsearch 7 sink.
{quote}
I had in mind that the DemultiplexingSink implements the unified Sink interface
and delegates to the given sinks based on the router. We can start simple and
do not use a separate operator topology but use only one sink topology.
{quote}
As a follow up to the previous one, I'd imagine this demultiplexer could
conceivably route records to multiple different classes of Sink (e.g. send
records to Elasticsearch, send records to Kafka, JDBC, etc.). Is it out of the
question for a single record to be sent to multiple different sinks (send to
all n different routes it knows about)? If so, I'd imagine that the
genericizing of the Sink type parameters might be tricky? Maybe not?
{quote}
Honestly, I did not think of these use cases before. It still feels a bit hard
to imagine the full picture of why someone would use the DemultplexingSink with
different sink implementations. I would start with the assumption that the
Sinks have the same type to simplify the initial setup. We can later extend it
if we see the necessity.
> Introduce DemultiplexingSink to Support Dynamic Sink Routing
> ------------------------------------------------------------
>
> Key: FLINK-24493
> URL: https://issues.apache.org/jira/browse/FLINK-24493
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
> Reporter: Rion Williams
> Assignee: Rion Williams
> Priority: Major
>
> Recently, FLINK-23977 attempted to introduce an approach to supporting
> dynamic routing for the Elasticsearch sink, however during some discussion
> within [the pull request|https://github.com/apache/flink/pull/17061], the
> idea was introduced to create a more generic approach.
> The idea being that we could introduce a common DemultiplexingSink and
> related interface for handling routing to any number of existing sinks
> similar to the implementation mentioned in FLINK-23977 to the common
> connectors directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
