[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714491#comment-16714491
]
vinoyang commented on FLINK-9461:
-
I created a new issue FLINK-8 based on FLIP-28 to track the work about
refactoring row time extractor.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711081#comment-16711081
]
vinoyang commented on FLINK-9461:
-
[~twalthr] Some interfaces, such as StreamTableSinkFactory, are used by
multiple connectors. So while migrating, I probably won't delete them, but
instead create a copy of the Java implementation in table-common. I have done
[most of the
work|https://github.com/yanghua/flink/commit/03e69684d7f584618fb5db04be13c63a403d6d12]
right now, but the transitive dependencies of some interfaces make it
impossible to compile successfully and need to rethink.
In addition, I suggest that we create an umbrella issue for migrating all
connectors and then use the current issue as a sub task.
The migration of connectors and the transformation of interfaces should be
centralized.
What do you think?
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700596#comment-16700596
]
Timo Walther commented on FLINK-9461:
-
I'm about to create a FLIP for the new module structure of {{flink-table}} as
[discussed in the design
document|https://docs.google.com/document/d/1PPo6goW7tOwxmpFuvLSjFnx7BF8IVz0w3dcmPPyqvoY/edit].
We can then refer people to the FLIP. But I'm also fine with mentioning again
in some dedicated dependency page.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700562#comment-16700562
]
Timo Walther commented on FLINK-9461:
-
[~StephanEwen] What do you mean with "handle connectors"?
The long-term goal is that connectors and formats only depend on
{{flink-table-common}} which depends on {{flink-core}} only (no further
transitive dependencies).
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700567#comment-16700567
]
Stephan Ewen commented on FLINK-9461:
-
I think it would be great to have a section in either the wiki or the internal
docs to outline exactly that, so that all connector contributors are aware of
this.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700534#comment-16700534
]
Stephan Ewen commented on FLINK-9461:
-
[~twalthr] Is there a plan on how to handle connectors across DataSet /
DataStream / Table API in general?
If not, let's make sure we have that so all connectors follow the same
dependency structure.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700357#comment-16700357
]
Timo Walther commented on FLINK-9461:
-
We can remove the deprecated methods and classes now. This can be done in a
separate PR.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700341#comment-16700341
]
vinoyang commented on FLINK-9461:
-
[~twalthr] To confirm a question, are the classes or methods marked by
"@Deprecated" included in our migration plan? Or can we remove it without
compatibility? For example, "StreamRecordTimestamp" depends on many classes,
but it is only in KafkaTableSourceBase.Builder (a static inner class labeled
@Deprecated).
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700102#comment-16700102
]
Timo Walther commented on FLINK-9461:
-
[~yanghua] Yes, this is why we only ported format-related classes so far. There
is a long tail of classes that are pulled in transitively. We can only change
that by modifying the interface for defining rowtime time attributes. I'm
pretty sure that we have to change the interfaces as part of the [connector
improvements|https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit#heading=h.41fd6rs7b3cf].
You can open a separate issue for this.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700074#comment-16700074
]
vinoyang commented on FLINK-9461:
-
[~twalthr] Here are some of the dependencies on _expression_ and _plan_
package.
+*StreamRecordTimestamp -> Expression*+
+*StreamRecordTimestamp -> ResolvedFieldReference -> Attribute*+
I think introducing it to "flink-table-common" is not what we expected.
It feels a bit tricky to deal with them.
What do you think of how to deal with them?
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697054#comment-16697054
]
Timo Walther commented on FLINK-9461:
-
Yes, this is the way to go. Feel free to start porting interfaces that
logically belong into the a common module according to the design document.
Let's discuss again if you see more and more classes are pulled in
transitively. For example, we cannot port \{{FilterableTableSource}} but I
think Kafka is not using it.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696693#comment-16696693
]
vinoyang commented on FLINK-9461:
-
hi [~twalthr] Thank you for your explanation of the context.
I should confirm with you to make sure I have not misunderstood what you mean.
Whether we can migrate the classes to flink-table-common, which are originally
in flink-table but are dependent on flink*_-_*connector*_-_*kafka, such as
StreamTableSource. Then make flink-connector-kafka directly dependent on
flink-table-common instead of flink-table. right?
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696589#comment-16696589
]
Timo Walther commented on FLINK-9461:
-
[~yanghua] The main work on this issue has already been started.
\{{flink-json}} is only used by deprecated code and can be removed once we drop
format specific sources and sinks. I think we can remove `KafkaJsonTableSink`
and `KafkaAvroTableSink` in the next release. Feel free to open a PR for this.
I had a private discussion with [~aljoscha] and [~StephanEwen] about how we
want to deal with \{{flink-table}} dependencies in connectors and formats. We
decided against dedicated \{{flink-table-connectors}} and
\{{flink-table-formats}} modules. Instead we started introducing a lightweight
\{{flink-table-common}} module that currently only formats use but connectors
can use this as well soon. See also the [discussion
here|https://docs.google.com/document/d/1PPo6goW7tOwxmpFuvLSjFnx7BF8IVz0w3dcmPPyqvoY/edit#].
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696503#comment-16696503
]
vinoyang commented on FLINK-9461:
-
Hi all,
The discussion of this issue does not seem to have a conclusion. Not long ago,
I just participated in the features of modern Kafka 2.0 and created a new
module: flink-connector-kafka. From the history of the discussion, [~fhueske]
's point of view seems to think that the flink table API should be treated
peer-to-peer with the flink datastream/dataset API. I prefer the table API to
be an upper layer lib. When the user only uses the DataStream API and the Kafka
connector, it does not have to depend on the flink-table.
I suggest that for this case, we introduce a parent module and a submodule,
such as "flink-table-source-sink/flink-table-source-sink-kafka", but there
seems to be a problem here, "flink-table-source-sink" seems to be a sub-module
of "flink-libraries". Once this is done, a three-tier structure will be
introduced, which will break the convention of the current flink project.
I hope to hear your thoughts. cc [~till.rohrmann] [~Zentol] [~twalthr] and
[~pnowojski]
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.8.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495248#comment-16495248
]
Till Rohrmann commented on FLINK-9461:
--
If {{flink-table}} would only contain some interfaces defining the sources,
sinks and operators, then it would not be a big problem. But this is obviously
not the case. The point I'm trying to make is that it feels a bit odd that a
Flink connector depends on a Flink library while it would be so simple to get
rid of it by introducing a dedicated module.
Assume that we might get rid of the Scala dependency in {{flink-runtime}}, then
we could still not remove the Scala suffix from the connectors because they
depend on {{flink-table}}.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495233#comment-16495233
]
Fabian Hueske commented on FLINK-9461:
--
Why should the API depend on the connector? All our connectors are tied to an
API and the Table API / SQL is an API just like the DataStream or DataSet API.
Should the DataStream API depend on Kafka (and Kinesis, Cassandra, ES, ...)?
The API provides an interface and the connectors implement the interface. We
followed that pattern for all APIs and connectors.
Moving all connector-related API classes to {{flink-core}} as suggested by
[~Zentol] would require a larger refactoring because some API classes reference
core classes of the Table API (implemented in Scala).
I'm not opposed to move connectors into individual modules but IMO, the
question is whether a few additional classes in a JAR file justify fragmenting
the connectors into API specific modules.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495170#comment-16495170
]
Till Rohrmann commented on FLINK-9461:
--
Logically, it does not make sense to couple a connector with its user (here
{{flink-table}}). If it all, then it the user should depend on it or ideally we
have a common abstraction/interface living for example in
{{flink-table-connectors}} which can be implemented by a concrete connector and
dropped in dynamically.
Not sure whether I buy the argument with too many maven modules. What would be
the problem with that? It rather looks to me that making
{{flink-connector-kafka}} depend on {{flink-table}} was a kind of short cut.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493324#comment-16493324
]
Chesnay Schepler commented on FLINK-9461:
-
IMO the dependency on flink-table isn't the core problem here but just a
symptom. As far as I'm concerned libraries and connectors should never have
dependencies on each other. The primitives required by connectors/libraries
should reside in the core of flink (in this case i guess the API modules or
flink-core).
This what we should look into is moving the table classes used outside of
flink-table into the core of Flink; we'Ve done this before with the {{Row}}
type.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493214#comment-16493214
]
Fabian Hueske commented on FLINK-9461:
--
I think this (and similar) issues should be discussed on the dev mailing list.
Decoupling all connectors (and formats) from {{flink-table}} means replicating
all connector modules, i.e., easily 10 more Maven modules.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9461) Disentangle flink-connector-kafka from flink-table and flink-json
[
https://issues.apache.org/jira/browse/FLINK-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493194#comment-16493194
]
Chesnay Schepler commented on FLINK-9461:
-
The cassandra connector has the same issue.
> Disentangle flink-connector-kafka from flink-table and flink-json
> -
>
> Key: FLINK-9461
> URL: https://issues.apache.org/jira/browse/FLINK-9461
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.6.0
>
>
> Currently, the {{flink-connector-kafka}} module has a dependency on
> {{flink-table}} and {{flink-json}}. The reason seems to be that the module
> contains the {{KafkaJsonTableSource}} and {{KafkaJsonTableSink}}. Even though
> the {{flink-table}} and {{flink-json}} dependency are marked as optional, the
> {{flink-connector-kafka}} will still contain the table sources and sinks. I
> think this is not a clean design.
> I would propose to move the table sources and sinks into a dedicated module
> which depends on {{flink-connector-kafka}}. That way we would better separate
> dependencies and could remove {{flink-table}} and {{flink-json}} from
> {{flink-connector-kafka}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
