[
https://issues.apache.org/jira/browse/FLINK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428994#comment-17428994
]
Rion Williams edited comment on FLINK-24493 at 10/14/21, 7:52 PM:
------------------------------------------------------------------
[~fabian.paul] / [~dmvk] / [~arvid]
I had a bit of time to think about this earlier today and I'm excited about
working on it. I did have a handful of questions that one or more of you folks
might be able to help with:
* I was curious if there was any documentation / examples / tests that I could
use to glean from with regards to the new Sink interface. I'm not terribly
familiar with it as much as the previous common uses of SinkFunctions, etc.
Could anyone point me in a direction on that front?
* Given the changes made to the Sink interface, would the expectation be that
this new DemultiplexingSink inherits from a RichSinkFunction similar to my
previous implementation for the use-case for Elasticsearch? Or would it just
directly implement the Sink interface and the router would govern how to
construct the sink itself [similar to the port of the Elasticsearch 7
sink|https://github.com/apache/flink/pull/17363/files].
* As a follow up to the previous one, I'd imagine this demultiplexer could
conceivably route records to multiple different classes of Sink (e.g. send
records to Elasticsearch, send records to Kafka, JDBC, etc.). Is it out of the
question for a single record to be sent to multiple different sinks (send to
all n different routes it knows about)? If so, I'd imagine that the
genericizing of the Sink type parameters might be tricky? Maybe not?
I'll keep digging around a bit - but I think after looking over the ES7
implementation (while writing this) might answer most of these questions since
it seems to easily show how the older interfaces compare to the new one (e.g.
SinkFunction.invoke() vs Sink.emitter).
Thanks folks! I'm hoping to dig to this a bit more next week as I'll be
participating in the trainings offered at Flink Forward and looking forward to
the sessions at the conference itself.
Rion
(P.S. How would one get a Flink shirt if they exist?)
was (Author: rionmonster):
[~fabian.paul] / [~dmvk] / [~arvid]
I had a bit of time to think about this earlier today and I'm excited about
working on it. I did have a handful of questions that one or more of you folks
might be able to help with:
* I was curious if there was any documentation / examples / tests that I could
use to glean from with regards to the new Sink interface. I'm not terribly
familiar with it as much as the previous common uses of SinkFunctions, etc.
Could anyone point me in a direction on that front?
* Given the changes made to the Sink interface, would the expectation be that
this new DemultiplexingSink inherits from a RichSinkFunction similar to my
previous implementation for the use-case for Elasticsearch? Or would it just
directly implement the Sink interface and the router would govern how to
construct the sink itself [similar to the port of the Elasticsearch 7
sink|https://github.com/apache/flink/pull/17363/files].
I'll keep digging around a bit - but I think after looking over the ES7
implementation (while writing this) might answer most of these questions since
it seems to easily show how the older interfaces compare to the new one (e.g.
SinkFunction.invoke() vs Sink.emitter).
Thanks folks! I'm hoping to dig to this a bit more next week as I'll be
participating in the trainings offered at Flink Forward and looking forward to
the sessions at the conference itself.
Rion
(P.S. How would one get a Flink shirt if they exist?)
> Introduce DemultiplexingSink to Support Dynamic Sink Routing
> ------------------------------------------------------------
>
> Key: FLINK-24493
> URL: https://issues.apache.org/jira/browse/FLINK-24493
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
> Reporter: Rion Williams
> Assignee: Rion Williams
> Priority: Major
>
> Recently, FLINK-23977 attempted to introduce an approach to supporting
> dynamic routing for the Elasticsearch sink, however during some discussion
> within [the pull request|https://github.com/apache/flink/pull/17061], the
> idea was introduced to create a more generic approach.
> The idea being that we could introduce a common DemultiplexingSink and
> related interface for handling routing to any number of existing sinks
> similar to the implementation mentioned in FLINK-23977 to the common
> connectors directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
