The root of the previous error seemed to be the flink version the connector was compiled for. I've tried compiling my own postgresql-cdc connector, but still have some issues with dependencies.
On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí <sebasma...@gmail.com> wrote: > Applied that parameter and that seems to get me some progress here. > > I still get the shade overlapping classes warning, but I get the > PostgreSQLTableFactory in the merged table.factories.Factory service file. > > However, now on runtime the application fails to find the debezium source > function class coming down to this error: > > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: > Cannot load user class: > com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction > ClassLoader info: URL ClassLoader: > Class not resolvable through given classloader. > > The class is indeed in jar, though. > > Any thougths? > > On Thu, 28 Jan 2021 at 09:57, Jark Wu <imj...@gmail.com> wrote: > >> Hi Sebastián, >> >> Could you try to add combine.children="append" attribute to the >> transformers configuration? >> You can also see the full shade plugin configuration here [1]. >> >> Best, >> Jark >> >> [1]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources >> >> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <sebasma...@gmail.com> >> wrote: >> >>> Hi Jark! >>> >>> Please find the full pom file attached. >>> >>> Best Regards, >>> >>> On Thu, 28 Jan 2021 at 03:21, Jark Wu <imj...@gmail.com> wrote: >>> >>>> Hi Sebastián, >>>> >>>> I think Dawid is right. >>>> >>>> Could you share the pom file? I also tried to >>>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and >>>> the Factory file contains >>>> >>>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory >>>> >>>> >>>> Best, >>>> Jark >>>> >>>> >>>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <sebasma...@gmail.com> >>>> wrote: >>>> >>>>> Thanks a lot for looking into it Dawid, >>>>> >>>>> In the >>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory >>>>> file I only see >>>>> >>>>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory >>>>> >>>>> Even after applying the ServicesResourceTransformer. >>>>> >>>>> >>>>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <dwysakow...@apache.org> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Unfortunately I am not familiar with the packaging of >>>>>> flink-connector-postgres-cdc. Maybe @Jark could help here? >>>>>> >>>>>> However, I think the problem that you cannot find the connector is >>>>>> caused because of lack of entry in the resulting Manifest file. If there >>>>>> are overlapping classes maven does not exclude whole dependencies, but >>>>>> rather picks the overlapping class from one of the two. Could you check >>>>>> if >>>>>> you see entries for all tables in >>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory. >>>>>> >>>>>> If not, you could try applying the ServicesResourceTransformer[1] >>>>>> >>>>>> Best, >>>>>> >>>>>> Dawid >>>>>> >>>>>> [1] >>>>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer >>>>>> On 26/01/2021 12:29, Sebastián Magrí wrote: >>>>>> >>>>>> Hi! >>>>>> >>>>>> I've reported an issue with the postgresql-cdc connector apparently >>>>>> caused by the maven shade plugin excluding either the JDBC connector or >>>>>> the >>>>>> cdc connector due to overlapping classes. The issue for reference is >>>>>> here: >>>>>> >>>>>> https://github.com/ververica/flink-cdc-connectors/issues/90 >>>>>> >>>>>> In the meantime, however, I've been trying to figure out if I can set >>>>>> up an exclusion rule to fix this in my pom.xml file, without success. >>>>>> >>>>>> The `org.postgresql:postgresql` dependency is being added manually by >>>>>> me to have a sink on a postgresql table and injected by the cdc connector >>>>>> seemingly via its debezium connector dependency. >>>>>> >>>>>> Any guidance or hints I could follow would be really appreciated. >>>>>> >>>>>> -- >>>>>> Sebastián Ramírez Magrí >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Sebastián Ramírez Magrí >>>>> >>>> >>> >>> -- >>> Sebastián Ramírez Magrí >>> >> > > -- > Sebastián Ramírez Magrí > -- Sebastián Ramírez Magrí
