Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-02-05 Thread Robert Metzger
I don't know what your dependency issue is (post it here if you want
help!), but I generally recommend using mvn dependency:tree to debug
version clashes (and then pin or exclude versions)

On Tue, Feb 2, 2021 at 9:23 PM Sebastián Magrí  wrote:

> The root of the previous error seemed to be the flink version the
> connector was compiled for. I've tried compiling my own postgresql-cdc
> connector, but still have some issues with dependencies.
>
> On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí 
> wrote:
>
>> Applied that parameter and that seems to get me some progress here.
>>
>> I still get the shade overlapping classes warning, but I get the
>> PostgreSQLTableFactory in the merged table.factories.Factory service file.
>>
>> However, now on runtime the application fails to find the debezium source
>> function class coming down to this error:
>>
>> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
>> Cannot load user class:
>> com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
>> ClassLoader info: URL ClassLoader:
>> Class not resolvable through given classloader.
>>
>> The class is indeed in jar, though.
>>
>> Any thougths?
>>
>> On Thu, 28 Jan 2021 at 09:57, Jark Wu  wrote:
>>
>>> Hi Sebastián,
>>>
>>> Could you try to add combine.children="append" attribute to the
>>> transformers configuration?
>>> You can also see the full shade plugin configuration here [1].
>>>
>>> Best,
>>> Jark
>>>
>>> [1]:
>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources
>>>
>>> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí 
>>> wrote:
>>>
 Hi Jark!

 Please find the full pom file attached.

 Best Regards,

 On Thu, 28 Jan 2021 at 03:21, Jark Wu  wrote:

> Hi Sebastián,
>
> I think Dawid is right.
>
> Could you share the pom file? I also tried to
> package flink-connector-postgres-cdc with ServicesResourceTransformer, and
> the Factory file contains
>
> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory
>
>
> Best,
> Jark
>
>
> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí 
> wrote:
>
>> Thanks a lot for looking into it Dawid,
>>
>> In the
>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
>> file I only see
>>
>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>>
>> Even after applying the ServicesResourceTransformer.
>>
>>
>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <
>> dwysakow...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> Unfortunately I am not familiar with the packaging of
>>> flink-connector-postgres-cdc. Maybe @Jark could help here?
>>>
>>> However, I think the problem that you cannot find the connector is
>>> caused because of lack of entry in the resulting Manifest file. If there
>>> are overlapping classes maven does not exclude whole dependencies, but
>>> rather picks the overlapping class from one of the two. Could you check 
>>> if
>>> you see entries for all tables in
>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>>>
>>> If not, you could try applying the ServicesResourceTransformer[1]
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1]
>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>>> On 26/01/2021 12:29, Sebastián Magrí wrote:
>>>
>>> Hi!
>>>
>>> I've reported an issue with the postgresql-cdc connector apparently
>>> caused by the maven shade plugin excluding either the JDBC connector or 
>>> the
>>> cdc connector due to overlapping classes. The issue for reference is 
>>> here:
>>>
>>> https://github.com/ververica/flink-cdc-connectors/issues/90
>>>
>>> In the meantime, however, I've been trying to figure out if I can
>>> set up an exclusion rule to fix this in my pom.xml file, without 
>>> success.
>>>
>>> The `org.postgresql:postgresql` dependency is being added manually
>>> by me to have a sink on a postgresql table and injected by the cdc
>>> connector seemingly via its debezium connector dependency.
>>>
>>> Any guidance or hints I could follow would be really appreciated.
>>>
>>> --
>>> Sebastián Ramírez Magrí
>>>
>>>
>>
>> --
>> Sebastián Ramírez Magrí
>>
>

 --
 Sebastián Ramírez Magrí

>>>
>>
>> --
>> Sebastián Ramírez Magrí
>>
>
>
> --
> Sebastián Ramírez Magrí
>


Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-02-02 Thread Sebastián Magrí
The root of the previous error seemed to be the flink version the connector
was compiled for. I've tried compiling my own postgresql-cdc connector, but
still have some issues with dependencies.

On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí  wrote:

> Applied that parameter and that seems to get me some progress here.
>
> I still get the shade overlapping classes warning, but I get the
> PostgreSQLTableFactory in the merged table.factories.Factory service file.
>
> However, now on runtime the application fails to find the debezium source
> function class coming down to this error:
>
> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
> Cannot load user class:
> com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
> ClassLoader info: URL ClassLoader:
> Class not resolvable through given classloader.
>
> The class is indeed in jar, though.
>
> Any thougths?
>
> On Thu, 28 Jan 2021 at 09:57, Jark Wu  wrote:
>
>> Hi Sebastián,
>>
>> Could you try to add combine.children="append" attribute to the
>> transformers configuration?
>> You can also see the full shade plugin configuration here [1].
>>
>> Best,
>> Jark
>>
>> [1]:
>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources
>>
>> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí 
>> wrote:
>>
>>> Hi Jark!
>>>
>>> Please find the full pom file attached.
>>>
>>> Best Regards,
>>>
>>> On Thu, 28 Jan 2021 at 03:21, Jark Wu  wrote:
>>>
 Hi Sebastián,

 I think Dawid is right.

 Could you share the pom file? I also tried to
 package flink-connector-postgres-cdc with ServicesResourceTransformer, and
 the Factory file contains

 com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory


 Best,
 Jark


 On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí 
 wrote:

> Thanks a lot for looking into it Dawid,
>
> In the
> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
> file I only see
>
> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>
> Even after applying the ServicesResourceTransformer.
>
>
> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz 
> wrote:
>
>> Hi,
>>
>> Unfortunately I am not familiar with the packaging of
>> flink-connector-postgres-cdc. Maybe @Jark could help here?
>>
>> However, I think the problem that you cannot find the connector is
>> caused because of lack of entry in the resulting Manifest file. If there
>> are overlapping classes maven does not exclude whole dependencies, but
>> rather picks the overlapping class from one of the two. Could you check 
>> if
>> you see entries for all tables in
>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>>
>> If not, you could try applying the ServicesResourceTransformer[1]
>>
>> Best,
>>
>> Dawid
>>
>> [1]
>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>> On 26/01/2021 12:29, Sebastián Magrí wrote:
>>
>> Hi!
>>
>> I've reported an issue with the postgresql-cdc connector apparently
>> caused by the maven shade plugin excluding either the JDBC connector or 
>> the
>> cdc connector due to overlapping classes. The issue for reference is 
>> here:
>>
>> https://github.com/ververica/flink-cdc-connectors/issues/90
>>
>> In the meantime, however, I've been trying to figure out if I can set
>> up an exclusion rule to fix this in my pom.xml file, without success.
>>
>> The `org.postgresql:postgresql` dependency is being added manually by
>> me to have a sink on a postgresql table and injected by the cdc connector
>> seemingly via its debezium connector dependency.
>>
>> Any guidance or hints I could follow would be really appreciated.
>>
>> --
>> Sebastián Ramírez Magrí
>>
>>
>
> --
> Sebastián Ramírez Magrí
>

>>>
>>> --
>>> Sebastián Ramírez Magrí
>>>
>>
>
> --
> Sebastián Ramírez Magrí
>


-- 
Sebastián Ramírez Magrí


Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-01-28 Thread Sebastián Magrí
Applied that parameter and that seems to get me some progress here.

I still get the shade overlapping classes warning, but I get the
PostgreSQLTableFactory in the merged table.factories.Factory service file.

However, now on runtime the application fails to find the debezium source
function class coming down to this error:

Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
Cannot load user class:
com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.

The class is indeed in jar, though.

Any thougths?

On Thu, 28 Jan 2021 at 09:57, Jark Wu  wrote:

> Hi Sebastián,
>
> Could you try to add combine.children="append" attribute to the
> transformers configuration?
> You can also see the full shade plugin configuration here [1].
>
> Best,
> Jark
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources
>
> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí 
> wrote:
>
>> Hi Jark!
>>
>> Please find the full pom file attached.
>>
>> Best Regards,
>>
>> On Thu, 28 Jan 2021 at 03:21, Jark Wu  wrote:
>>
>>> Hi Sebastián,
>>>
>>> I think Dawid is right.
>>>
>>> Could you share the pom file? I also tried to
>>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and
>>> the Factory file contains
>>>
>>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory
>>>
>>>
>>> Best,
>>> Jark
>>>
>>>
>>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí 
>>> wrote:
>>>
 Thanks a lot for looking into it Dawid,

 In the
 src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
 file I only see

 org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

 Even after applying the ServicesResourceTransformer.


 On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz 
 wrote:

> Hi,
>
> Unfortunately I am not familiar with the packaging of
> flink-connector-postgres-cdc. Maybe @Jark could help here?
>
> However, I think the problem that you cannot find the connector is
> caused because of lack of entry in the resulting Manifest file. If there
> are overlapping classes maven does not exclude whole dependencies, but
> rather picks the overlapping class from one of the two. Could you check if
> you see entries for all tables in
> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>
> If not, you could try applying the ServicesResourceTransformer[1]
>
> Best,
>
> Dawid
>
> [1]
> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
> On 26/01/2021 12:29, Sebastián Magrí wrote:
>
> Hi!
>
> I've reported an issue with the postgresql-cdc connector apparently
> caused by the maven shade plugin excluding either the JDBC connector or 
> the
> cdc connector due to overlapping classes. The issue for reference is here:
>
> https://github.com/ververica/flink-cdc-connectors/issues/90
>
> In the meantime, however, I've been trying to figure out if I can set
> up an exclusion rule to fix this in my pom.xml file, without success.
>
> The `org.postgresql:postgresql` dependency is being added manually by
> me to have a sink on a postgresql table and injected by the cdc connector
> seemingly via its debezium connector dependency.
>
> Any guidance or hints I could follow would be really appreciated.
>
> --
> Sebastián Ramírez Magrí
>
>

 --
 Sebastián Ramírez Magrí

>>>
>>
>> --
>> Sebastián Ramírez Magrí
>>
>

-- 
Sebastián Ramírez Magrí


Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-01-28 Thread Jark Wu
Hi Sebastián,

Could you try to add combine.children="append" attribute to the
transformers configuration?
You can also see the full shade plugin configuration here [1].

Best,
Jark

[1]:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources

On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí  wrote:

> Hi Jark!
>
> Please find the full pom file attached.
>
> Best Regards,
>
> On Thu, 28 Jan 2021 at 03:21, Jark Wu  wrote:
>
>> Hi Sebastián,
>>
>> I think Dawid is right.
>>
>> Could you share the pom file? I also tried to
>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and
>> the Factory file contains
>>
>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory
>>
>>
>> Best,
>> Jark
>>
>>
>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí 
>> wrote:
>>
>>> Thanks a lot for looking into it Dawid,
>>>
>>> In the
>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
>>> file I only see
>>>
>>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>>>
>>> Even after applying the ServicesResourceTransformer.
>>>
>>>
>>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz 
>>> wrote:
>>>
 Hi,

 Unfortunately I am not familiar with the packaging of
 flink-connector-postgres-cdc. Maybe @Jark could help here?

 However, I think the problem that you cannot find the connector is
 caused because of lack of entry in the resulting Manifest file. If there
 are overlapping classes maven does not exclude whole dependencies, but
 rather picks the overlapping class from one of the two. Could you check if
 you see entries for all tables in
 src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

 If not, you could try applying the ServicesResourceTransformer[1]

 Best,

 Dawid

 [1]
 https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
 On 26/01/2021 12:29, Sebastián Magrí wrote:

 Hi!

 I've reported an issue with the postgresql-cdc connector apparently
 caused by the maven shade plugin excluding either the JDBC connector or the
 cdc connector due to overlapping classes. The issue for reference is here:

 https://github.com/ververica/flink-cdc-connectors/issues/90

 In the meantime, however, I've been trying to figure out if I can set
 up an exclusion rule to fix this in my pom.xml file, without success.

 The `org.postgresql:postgresql` dependency is being added manually by
 me to have a sink on a postgresql table and injected by the cdc connector
 seemingly via its debezium connector dependency.

 Any guidance or hints I could follow would be really appreciated.

 --
 Sebastián Ramírez Magrí


>>>
>>> --
>>> Sebastián Ramírez Magrí
>>>
>>
>
> --
> Sebastián Ramírez Magrí
>


Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-01-28 Thread Sebastián Magrí
Hi Jark!

Please find the full pom file attached.

Best Regards,

On Thu, 28 Jan 2021 at 03:21, Jark Wu  wrote:

> Hi Sebastián,
>
> I think Dawid is right.
>
> Could you share the pom file? I also tried to
> package flink-connector-postgres-cdc with ServicesResourceTransformer, and
> the Factory file contains
>
> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory
>
>
> Best,
> Jark
>
>
> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí 
> wrote:
>
>> Thanks a lot for looking into it Dawid,
>>
>> In the
>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
>> file I only see
>>
>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>>
>> Even after applying the ServicesResourceTransformer.
>>
>>
>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz 
>> wrote:
>>
>>> Hi,
>>>
>>> Unfortunately I am not familiar with the packaging of
>>> flink-connector-postgres-cdc. Maybe @Jark could help here?
>>>
>>> However, I think the problem that you cannot find the connector is
>>> caused because of lack of entry in the resulting Manifest file. If there
>>> are overlapping classes maven does not exclude whole dependencies, but
>>> rather picks the overlapping class from one of the two. Could you check if
>>> you see entries for all tables in
>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>>>
>>> If not, you could try applying the ServicesResourceTransformer[1]
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1]
>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>>> On 26/01/2021 12:29, Sebastián Magrí wrote:
>>>
>>> Hi!
>>>
>>> I've reported an issue with the postgresql-cdc connector apparently
>>> caused by the maven shade plugin excluding either the JDBC connector or the
>>> cdc connector due to overlapping classes. The issue for reference is here:
>>>
>>> https://github.com/ververica/flink-cdc-connectors/issues/90
>>>
>>> In the meantime, however, I've been trying to figure out if I can set up
>>> an exclusion rule to fix this in my pom.xml file, without success.
>>>
>>> The `org.postgresql:postgresql` dependency is being added manually by me
>>> to have a sink on a postgresql table and injected by the cdc connector
>>> seemingly via its debezium connector dependency.
>>>
>>> Any guidance or hints I could follow would be really appreciated.
>>>
>>> --
>>> Sebastián Ramírez Magrí
>>>
>>>
>>
>> --
>> Sebastián Ramírez Magrí
>>
>

-- 
Sebastián Ramírez Magrí
http://maven.apache.org/POM/4.0.0;
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;>
	4.0.0

	org.sebasmagri
	stats-aggregator
	1.0-SNAPSHOT
	jar

	Stats Aggregator

	
		UTF-8
		1.12.1
		2.11
		2.11.12
		2.12.1
		42.2.18.jre7
	

	
		
			org.apache.flink
			flink-scala_${scala.binary.version}
			${flink.version}
			provided
		
		
			org.apache.flink
			flink-streaming-scala_${scala.binary.version}
			${flink.version}
			provided
		
		
			org.apache.flink
			flink-clients_${scala.binary.version}
			${flink.version}
			provided
		

		
		
			org.scala-lang
			scala-library
			${scala.version}
			provided
		

		

		
		
			com.alibaba.ververica
			flink-connector-postgres-cdc
			1.1.0
		
		
			org.apache.flink
			flink-connector-jdbc_${scala.binary.version}
			${flink.version}
		
		
			org.postgresql
			postgresql
			${postgresql.version}
		

		
		
		
			org.apache.logging.log4j
			log4j-slf4j-impl
			${log4j.version}
			runtime
		
		
			org.apache.logging.log4j
			log4j-api
			${log4j.version}
			runtime
		
		
			org.apache.logging.log4j
			log4j-core
			${log4j.version}
			runtime
		

		

		
			org.apache.flink
			flink-table-api-scala-bridge_${scala.binary.version}
			${flink.version}
			provided
		
		
			org.apache.flink
			flink-table-planner-blink_${scala.binary.version}
			${flink.version}
			provided
		
		
			org.apache.flink
			flink-streaming-scala_${scala.binary.version}
			${flink.version}
			provided
		

	

	
		
			
			
			
org.apache.maven.plugins
maven-shade-plugin
3.1.1

	
	
		package
		
			shade
		
		
			
			

	org.apache.flink:force-shading
	com.google.code.findbugs:jsr305
	org.slf4j:*
	org.apache.logging.log4j:*

			
			

	
	*:*
	
		META-INF/*.SF
		META-INF/*.DSA
		META-INF/*.RSA
	

			
			


	org.sebasmagri.StatsAggregator

			
		
	

			

			
			
org.apache.maven.plugins
maven-compiler-plugin
3.1

	1.8
	1.8

			

			
			
net.alchim31.maven
scala-maven-plugin
3.2.2

	
		
			compile
			testCompile
		
	


	
		-nobootcp
	

			

			
			
org.apache.maven.plugins

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-01-27 Thread Jark Wu
Hi Sebastián,

I think Dawid is right.

Could you share the pom file? I also tried to
package flink-connector-postgres-cdc with ServicesResourceTransformer, and
the Factory file contains

com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory


Best,
Jark


On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí  wrote:

> Thanks a lot for looking into it Dawid,
>
> In the
> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
> file I only see
>
> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>
> Even after applying the ServicesResourceTransformer.
>
>
> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz 
> wrote:
>
>> Hi,
>>
>> Unfortunately I am not familiar with the packaging of
>> flink-connector-postgres-cdc. Maybe @Jark could help here?
>>
>> However, I think the problem that you cannot find the connector is caused
>> because of lack of entry in the resulting Manifest file. If there are
>> overlapping classes maven does not exclude whole dependencies, but rather
>> picks the overlapping class from one of the two. Could you check if you see
>> entries for all tables in
>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>>
>> If not, you could try applying the ServicesResourceTransformer[1]
>>
>> Best,
>>
>> Dawid
>>
>> [1]
>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>> On 26/01/2021 12:29, Sebastián Magrí wrote:
>>
>> Hi!
>>
>> I've reported an issue with the postgresql-cdc connector apparently
>> caused by the maven shade plugin excluding either the JDBC connector or the
>> cdc connector due to overlapping classes. The issue for reference is here:
>>
>> https://github.com/ververica/flink-cdc-connectors/issues/90
>>
>> In the meantime, however, I've been trying to figure out if I can set up
>> an exclusion rule to fix this in my pom.xml file, without success.
>>
>> The `org.postgresql:postgresql` dependency is being added manually by me
>> to have a sink on a postgresql table and injected by the cdc connector
>> seemingly via its debezium connector dependency.
>>
>> Any guidance or hints I could follow would be really appreciated.
>>
>> --
>> Sebastián Ramírez Magrí
>>
>>
>
> --
> Sebastián Ramírez Magrí
>


Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-01-26 Thread Sebastián Magrí
Thanks a lot for looking into it Dawid,

In the
src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz 
wrote:

> Hi,
>
> Unfortunately I am not familiar with the packaging of
> flink-connector-postgres-cdc. Maybe @Jark could help here?
>
> However, I think the problem that you cannot find the connector is caused
> because of lack of entry in the resulting Manifest file. If there are
> overlapping classes maven does not exclude whole dependencies, but rather
> picks the overlapping class from one of the two. Could you check if you see
> entries for all tables in
> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>
> If not, you could try applying the ServicesResourceTransformer[1]
>
> Best,
>
> Dawid
>
> [1]
> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
> On 26/01/2021 12:29, Sebastián Magrí wrote:
>
> Hi!
>
> I've reported an issue with the postgresql-cdc connector apparently caused
> by the maven shade plugin excluding either the JDBC connector or the cdc
> connector due to overlapping classes. The issue for reference is here:
>
> https://github.com/ververica/flink-cdc-connectors/issues/90
>
> In the meantime, however, I've been trying to figure out if I can set up
> an exclusion rule to fix this in my pom.xml file, without success.
>
> The `org.postgresql:postgresql` dependency is being added manually by me
> to have a sink on a postgresql table and injected by the cdc connector
> seemingly via its debezium connector dependency.
>
> Any guidance or hints I could follow would be really appreciated.
>
> --
> Sebastián Ramírez Magrí
>
>

-- 
Sebastián Ramírez Magrí


Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-01-26 Thread Dawid Wysakowicz
Hi,

Unfortunately I am not familiar with the packaging of
flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is
caused because of lack of entry in the resulting Manifest file. If there
are overlapping classes maven does not exclude whole dependencies, but
rather picks the overlapping class from one of the two. Could you check
if you see entries for all tables in
src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1]
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
> Hi!
>
> I've reported an issue with the postgresql-cdc connector apparently
> caused by the maven shade plugin excluding either the JDBC connector
> or the cdc connector due to overlapping classes. The issue for
> reference is here:
>
> https://github.com/ververica/flink-cdc-connectors/issues/90
>
> In the meantime, however, I've been trying to figure out if I can set
> up an exclusion rule to fix this in my pom.xml file, without success.
>
> The `org.postgresql:postgresql` dependency is being added manually by
> me to have a sink on a postgresql table and injected by the cdc
> connector seemingly via its debezium connector dependency.
>
> Any guidance or hints I could follow would be really appreciated.
>
> -- 
> Sebastián Ramírez Magrí


signature.asc
Description: OpenPGP digital signature


Conflicts between the JDBC and postgresql-cdc SQL connectors

2021-01-26 Thread Sebastián Magrí
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused
by the maven shade plugin excluding either the JDBC connector or the cdc
connector due to overlapping classes. The issue for reference is here:

https://github.com/ververica/flink-cdc-connectors/issues/90

In the meantime, however, I've been trying to figure out if I can set up an
exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to
have a sink on a postgresql table and injected by the cdc connector
seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

-- 
Sebastián Ramírez Magrí