[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743256#comment-17743256
]
Yun Gao commented on FLINK-32455:
-
Hi [~renqs] [~tzulitai] sorry for the late reply, it might takes some time to
finish the above fix, thus I think it would be indeed preferred to first
merging the hotfix. I'll try to finish the fix as soon as possible.
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: kafka-3.0.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: kafka-3.0.1, kafka-3.1.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742413#comment-17742413
]
Tzu-Li (Gordon) Tai commented on FLINK-32455:
-
Merged to flink-connector-kafka via:
main - 21d3b10e4d66aca82d70a05a4ab24fb5cf2db348
v3.0 - 59ac7389fde0200dbd6b928cf1e9fcb8c29b6350
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741999#comment-17741999
]
Tzu-Li (Gordon) Tai commented on FLINK-32455:
-
[~renqs] I'm merging my hotfix PR now which should unblock things for the time
being. The PR has already been reviewed and approved by [~gaoyunhaii].
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741997#comment-17741997
]
Tzu-Li (Gordon) Tai commented on FLINK-32455:
-
[~gaoyunhaii] that plan sounds good to me. Are you planning to steps 1. and 2.
already for 1.18? If yes I can probably revert the "quick" fix PR I did in the
Flink Kafka connector repo.
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741895#comment-17741895
]
Qingsheng Ren commented on FLINK-32455:
---
[~gaoyunhaii] is there any progress on this issue? Thanks
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740024#comment-17740024
]
Yun Gao commented on FLINK-32455:
-
Thanks [~tzulitai] for the quick fix!
For the formal fix, I think we might also use the same option, namely we
# Revert the `TypeSerializerUpgradeTestBase` so the connectors libraries could
continue to use it.
# Introduce a new `MigratedTypeSerializerUpgradeTestBase` and make all the
tests inside flink library to use.
# Then after 1.18 get published, we could move
`MigratedTypeSerializerUpgradeTestBase` back to
`TypeSerializerUpgradeTestBase`, and also migrates the tests in the connector
libraries.
What do you think about this?
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739797#comment-17739797
]
Tzu-Li (Gordon) Tai commented on FLINK-32455:
-
FYI preview PR for the fast workaround fix:
https://github.com/apache/flink-connector-kafka/pull/39
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739787#comment-17739787
]
Tzu-Li (Gordon) Tai commented on FLINK-32455:
-
One (quick) fix for now might be to make a copy of the original
{{TypeSerializerUpgradeTestBase}} to the Kafka connector, and only afterwards
introduce a proper public-facing test utility in {{apache/flink}} and only then
migration the Kafka connector test code to move to that.
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739785#comment-17739785
]
Tzu-Li (Gordon) Tai commented on FLINK-32455:
-
[~gaoyunhaii] sorry for the slow reply - just finished other stuff and circling
back to this.
> refactor the existing migration tests to the new framework.
The main challenge here is that external connector repos generally need to
build against the latest 2 major version, i.e. for now that would be 1.17.x and
the upcoming 1.18.x version.
Therefore, we can't simply refactor the migration tests in the Kafka connector
to use the new {{MigrationTest}} abstraction, otherwise the code won't be able
to simultaneously build against older versions.
Specifically, the situation is that:
* The Kafka connector code has a test ({{{}KafkaSerializerUpgradeTest{}}})
that depends on {{{}TypeSerializerUpgradeTestBase{}}}, which with FLINK-27518
now implements the new {{{}MigrationTest{}}}.
* After implementing {{{}MigrationTest{}}}, {{TypeSerializerUpgradeTestBase}}
has a breaking change across versions 1.17.x and 1.18.x.
* Therefore, it is not possible to update the test code to simultaneously
build against both versions.
To move forward, I think we need to:
# We might need to revert the changes to middle-layer test util abstractions
like {{TypeSerializerUpgradeTestBase}} that externalized connector code might
be depending on.
# At the same time, formally introduce a replacement for them in the
{{fink-migration-test-utils}} package, and mark the original
{{TypeSerializerUpgradeTestBase}} as deprecated.
# Gradually migrate the test code in the Kafka connector to use
{{flink-migration-test-utils}} after a few connector releases.
What do you think?
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738503#comment-17738503
]
Yun Gao commented on FLINK-32455:
-
May I first have a double confirmation on the direction we'd like to go:
1. As a whole, FLINK-27518 aims to avoid generating new snapshots for
migration tests manually on publishing new version. Since there are also
migration tests in the Kafka repo, I think it is also useful to also introduce
the improve here.
2. for the fink-migration-test-utils dependency, now it requires the module
using migration tests to introduce the dependency directly, thus it should also
not be a blocker.
If we think it is ok to go this way, I could open a PR to refactor the existing
migration tests to the new framework.
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738502#comment-17738502
]
Yun Gao commented on FLINK-32455:
-
Sorry for missing the external repositories, I'll also have a double check
here.
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738003#comment-17738003
]
Matthias Pohl commented on FLINK-32455:
---
True, we missed that there's the externalization happening while we worked on
the test data generation. Sorry for that. [~tzulitai] ping me if you need help
and/or a review.
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737991#comment-17737991
]
Martijn Visser commented on FLINK-32455:
Also cc [~mapohl]
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-32455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737935#comment-17737935
]
Tzu-Li (Gordon) Tai commented on FLINK-32455:
-
cc [~gaoyunhaii]
> Breaking change in TypeSerializerUpgradeTestBase prevents
> flink-connector-kafka from building against 1.18-SNAPSHOT
> ---
>
> Key: FLINK-32455
> URL: https://issues.apache.org/jira/browse/FLINK-32455
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
> Fix For: 1.18.0
>
>
> FLINK-27518 introduced a breaking signature change to the abstract class
> {{TypeSerializerUpgradeTestBase}}, specifically the abstract
> {{createTestSpecifications}} method signature was changed. This breaks
> downstream test code in externalized connector repos, e.g.
> flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
> Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
> downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
